Oct 30 21:43:46.261947 ip-10-0-131-193 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:43:46.629438 ip-10-0-131-193 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:43:46.629438 ip-10-0-131-193 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:43:46.629438 ip-10-0-131-193 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:43:46.629438 ip-10-0-131-193 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:43:46.634263 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634141 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:43:46.634263 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634262 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:43:46.634263 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634266 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:43:46.634263 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634270 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634273 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634276 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634279 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634282 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634284 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634287 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634290 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634292 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634294 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634297 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634299 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634302 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634305 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634308 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634310 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634313 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634316 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634318 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:43:46.634414 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634321 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634329 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634332 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634334 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634337 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634340 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634342 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634345 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634347 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634350 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634352 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634355 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634358 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634360 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634364 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634367 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634370 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634373 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634375 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:43:46.634862 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634379 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634384 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634387 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634390 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634393 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634395 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634398 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634401 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634403 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634407 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634409 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634412 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634415 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634418 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634420 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634423 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634426 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634428 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634431 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:43:46.635305 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634434 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634436 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634439 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634444 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634449 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634452 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634455 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634458 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634461 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634465 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634468 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634471 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634473 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634476 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634479 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634481 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634484 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634487 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634489 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:43:46.635761 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634492 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634494 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634497 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634499 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634502 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634505 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634508 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634510 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634513 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634516 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634519 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634521 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634524 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634526 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634529 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634532 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634535 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634537 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634540 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634542 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:43:46.636243 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634545 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634548 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634551 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634553 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634556 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634558 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.634561 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635354 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635363 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635368 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635372 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635375 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635378 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635380 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635383 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635386 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635389 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635392 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635395 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:43:46.636724 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635398 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635401 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635403 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635406 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635409 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635411 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635414 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635416 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635419 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635422 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635424 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635428 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635431 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635433 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635436 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635439 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635442 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635444 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635447 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635450 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:43:46.637216 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635452 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635455 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635457 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635460 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635463 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635465 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635468 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635471 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635473 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635476 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635478 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635484 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635487 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635490 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635492 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635495 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635498 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635500 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635503 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635506 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:43:46.637714 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635509 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635511 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635514 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635516 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635519 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635521 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635524 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635533 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635536 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635538 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635541 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635543 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635546 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635549 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635552 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635554 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635557 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635559 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635562 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:43:46.638214 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635564 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635568 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635572 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635575 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635578 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635581 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635584 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635587 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635589 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635592 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635595 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635597 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635602 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635605 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635607 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635627 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635632 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635634 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635637 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:43:46.638672 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635639 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635644 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635647 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635649 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635652 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635654 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635657 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635659 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635662 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635664 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635667 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635670 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635672 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635674 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635677 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.635681 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636155 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636160 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636163 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:43:46.639121 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636166 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636169 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636171 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636174 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636177 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636179 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636197 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636201 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636204 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636207 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636210 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636214 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636217 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636220 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636234 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636237 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636240 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636243 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636245 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636248 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:43:46.639561 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636251 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636254 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636257 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636259 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636262 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636265 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636267 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636270 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636272 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636275 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636277 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636280 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636283 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636286 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636288 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636291 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636293 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636296 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636299 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:43:46.640054 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636302 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636304 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636307 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636310 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636312 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636315 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636317 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636322 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636326 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636329 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636332 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636335 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636338 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636341 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636344 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636346 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636349 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636351 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636354 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636357 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:43:46.640504 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636360 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636362 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636365 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636368 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636371 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636373 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636376 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636379 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636381 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636384 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636387 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636390 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636393 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636395 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636398 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636400 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636403 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636405 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636408 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636410 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:43:46.640988 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636413 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636415 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636418 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636420 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636423 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636425 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636428 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636431 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636434 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636436 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636439 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636442 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636444 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636447 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636449 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636453 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636455 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636458 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636460 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:43:46.641479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636463 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636466 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636468 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636471 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.636473 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637669 2576 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637678 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637686 2576 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637690 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637694 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637698 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637702 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637706 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637709 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637712 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637715 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637719 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637722 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637725 2576 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637728 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637731 2576 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637734 2576 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637737 2576 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:43:46.642032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637741 2576 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637744 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637746 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637750 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637753 2576 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637757 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637760 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637763 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637766 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637769 2576 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637772 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637775 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637782 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637786 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637790 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637793 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637796 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637799 2576 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637802 2576 flags.go:64] FLAG: --enable-server="true" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637805 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637810 2576 flags.go:64] FLAG: --event-burst="100" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637813 2576 flags.go:64] FLAG: --event-qps="50" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637816 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637819 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:43:46.642547 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637822 2576 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637827 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637830 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637833 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637836 2576 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637840 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637843 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637846 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637849 2576 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637852 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637854 2576 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637857 2576 flags.go:64] FLAG: --feature-gates="" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637862 2576 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637865 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637868 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637871 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637874 2576 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637877 2576 flags.go:64] FLAG: --help="false" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637880 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637884 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637890 2576 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637894 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637897 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:43:46.643147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637900 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637903 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637906 2576 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637909 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637912 2576 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637915 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637918 2576 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637921 2576 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637924 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637927 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637930 2576 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637933 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637936 2576 flags.go:64] FLAG: --lock-file="" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637938 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637942 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637945 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637951 2576 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637954 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637957 2576 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637960 2576 flags.go:64] FLAG: --logging-format="text" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637963 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637966 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637969 2576 flags.go:64] FLAG: --manifest-url="" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637973 2576 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637977 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:43:46.643728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637980 2576 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637984 2576 flags.go:64] FLAG: --max-pods="110" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637987 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637990 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637996 2576 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.637999 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638002 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638005 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638008 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638015 2576 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638019 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638021 2576 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638025 2576 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638027 2576 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638030 2576 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638033 2576 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638036 2576 flags.go:64] FLAG: --port="10250" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638039 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638042 2576 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-03edbd41cbd6ed55e" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638046 2576 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638049 2576 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638052 2576 flags.go:64] FLAG: --register-node="true" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638055 2576 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638059 2576 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638062 2576 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638065 2576 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:43:46.644336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638068 2576 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638072 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638075 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638078 2576 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638081 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638084 2576 flags.go:64] FLAG: --runonce="false" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638086 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638090 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638093 2576 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638096 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638101 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638105 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638108 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638111 2576 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638114 2576 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638117 2576 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638120 2576 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638123 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638126 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638129 2576 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638132 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638137 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638140 2576 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638143 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638147 2576 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:43:46.644970 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638150 2576 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638153 2576 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638156 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638159 2576 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638162 2576 flags.go:64] FLAG: --v="2" Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638166 2576 flags.go:64] FLAG: --version="false" Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638170 2576 flags.go:64] FLAG: --vmodule="" Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638174 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638177 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638316 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638321 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638324 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638327 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638329 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638332 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638334 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638337 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638343 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638345 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638349 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638352 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638354 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:43:46.645548 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638357 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638359 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638362 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638365 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638368 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638370 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638373 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638375 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638378 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638380 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638383 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638385 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638388 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638391 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638395 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638398 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638401 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638404 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638406 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:43:46.646092 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638409 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638411 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638414 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638416 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638419 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638421 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638424 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638426 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638431 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638434 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638437 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638440 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638443 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638446 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638448 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638451 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638453 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638456 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638459 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638461 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638464 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:43:46.646543 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638467 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638469 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638472 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638474 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638477 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638479 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638482 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638484 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638487 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638490 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638492 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638495 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638497 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638500 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638503 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638505 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638508 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638510 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638513 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:43:46.647154 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638518 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638521 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638523 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638526 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638529 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638532 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638534 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638537 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638539 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638542 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638545 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638547 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638550 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638552 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638555 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638557 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638560 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638562 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638565 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:43:46.647632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638569 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:43:46.648080 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638572 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:43:46.648080 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638575 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:43:46.648080 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638578 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:43:46.648080 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638581 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:43:46.648080 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638583 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:43:46.648080 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638586 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:43:46.648080 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638589 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:43:46.648080 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638591 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:43:46.648080 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638594 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:43:46.648080 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638597 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:43:46.648080 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638599 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:43:46.648080 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638602 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:43:46.648080 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638607 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:43:46.648080 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638626 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:43:46.648080 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.638655 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638690 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638695 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638702 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638705 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638708 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638710 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638713 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638715 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638718 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638720 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638723 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638726 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638728 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638731 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638733 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638736 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638738 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638741 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638744 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638746 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:43:46.648480 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638749 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638751 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638754 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638757 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638759 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638762 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638764 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638767 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638769 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638772 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638776 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638778 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638781 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638784 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638786 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638789 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638792 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638794 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638797 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:43:46.648987 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638799 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638802 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638804 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638807 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638809 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638811 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638814 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638816 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638819 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638821 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638824 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638826 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638829 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638832 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638835 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638837 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638839 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638842 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638845 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:43:46.649415 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638847 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:43:46.650380 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638849 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:43:46.650380 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638852 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:43:46.650380 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638856 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:43:46.650380 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638860 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:43:46.650380 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638863 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:43:46.650380 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638866 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:43:46.650380 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638868 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:43:46.650380 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638871 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:43:46.650380 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638874 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:43:46.650380 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638877 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:43:46.650380 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638879 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:43:46.650380 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638882 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:43:46.650380 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638884 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:43:46.650380 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638887 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:43:46.650380 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638890 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:43:46.650380 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638892 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:43:46.650380 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638895 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:43:46.650380 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638897 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638900 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638902 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638905 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638907 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638909 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638912 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638915 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638917 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638920 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638922 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638925 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638927 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638930 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638932 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638935 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638937 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638940 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638943 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638945 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:43:46.650905 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638948 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638950 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638953 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638955 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638958 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638961 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638963 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638967 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638971 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.638974 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.645669 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.645685 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.646426 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.646433 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646491 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646497 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646501 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646504 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646507 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646510 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:43:46.651360 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646513 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646516 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646518 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646522 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646525 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646528 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646531 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646533 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646536 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646539 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646541 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646544 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646547 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646550 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646553 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646556 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646559 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646562 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646565 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:43:46.651841 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646567 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646570 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646573 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646576 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646585 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646589 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646591 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646594 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646597 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646599 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646602 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646604 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646607 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646610 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646632 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646636 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646639 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646642 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646645 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646647 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:43:46.652301 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646650 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646653 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646656 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646658 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646661 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646663 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646666 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646669 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646672 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646674 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646677 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646680 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646682 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646685 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646687 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646690 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646693 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646703 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646706 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646709 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:43:46.652830 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646711 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646714 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646716 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646719 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646721 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646724 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646726 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646729 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646731 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646734 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646736 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646739 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646741 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646744 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646746 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646749 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646751 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646755 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646759 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646762 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:43:46.653355 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646765 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646768 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646771 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646773 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646776 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646778 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646781 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646784 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646787 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646789 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646797 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646800 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646802 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646805 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646808 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646810 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646814 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646817 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646821 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:43:46.653835 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646823 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:43:46.654296 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.646826 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:43:46.654296 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.646854 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:43:46.654296 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647033 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:43:46.654296 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647037 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:43:46.654296 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647040 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:43:46.654296 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647043 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:43:46.654296 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647046 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:43:46.654296 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647048 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:43:46.654296 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647051 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:43:46.654296 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647053 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:43:46.654296 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647056 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:43:46.654296 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647058 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:43:46.654296 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647061 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:43:46.654296 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647064 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:43:46.654296 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647066 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:43:46.654296 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647069 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647071 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647074 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647076 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647079 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647082 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647084 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647087 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647096 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647098 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647101 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647103 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647106 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647108 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647111 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647113 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647116 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647118 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647121 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647123 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:43:46.654759 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647126 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647128 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647131 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647133 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647136 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647138 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647141 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647143 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647146 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647148 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647151 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647155 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647159 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647162 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647165 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647169 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647172 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647174 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647177 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:43:46.655242 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647179 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647182 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647190 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647193 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647196 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647198 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647201 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647203 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647206 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647208 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647211 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647213 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647216 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647218 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647221 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647224 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647226 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647229 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647231 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647234 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:43:46.655708 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647236 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647238 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647241 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647244 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647252 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647255 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647257 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647260 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647262 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647265 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647268 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647270 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647273 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647275 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647277 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647286 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647288 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647291 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647294 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647296 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:43:46.656164 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647299 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:43:46.656657 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647301 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:43:46.656657 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647304 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:43:46.656657 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647307 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:43:46.656657 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647309 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:43:46.656657 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647312 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:43:46.656657 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647314 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:43:46.656657 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647317 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:43:46.656657 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647319 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:43:46.656657 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647321 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:43:46.656657 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647324 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:43:46.656657 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647327 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:43:46.656657 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647330 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:43:46.656657 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647334 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:43:46.656657 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.647356 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:43:46.656657 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647395 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647399 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647402 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647404 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647407 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647409 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647412 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647414 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647417 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647420 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647422 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647425 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647427 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647436 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647439 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647441 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647443 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647446 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647449 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:43:46.657014 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647451 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647454 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647456 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647459 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647461 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647464 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647466 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647469 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647471 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647474 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647477 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647479 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647481 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647488 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647490 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647493 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647496 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647498 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647501 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647503 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:43:46.657447 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647506 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647508 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647511 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647513 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647516 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647519 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647521 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647530 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647533 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647535 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647538 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647540 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647543 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647545 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647548 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647550 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647553 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647557 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647560 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:43:46.657939 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647563 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647566 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647568 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647573 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647575 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647578 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647582 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647584 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647587 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647589 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647592 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647595 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647597 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647600 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647602 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647605 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647607 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647625 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647629 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647632 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:43:46.658367 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647635 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647644 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647647 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647650 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647652 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647655 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647657 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647660 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647662 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647665 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647667 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647670 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647673 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647675 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647677 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647681 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647684 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647687 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647691 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647693 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:43:46.658853 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647696 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:43:46.659311 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647698 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:43:46.659311 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647701 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:43:46.659311 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647704 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:43:46.659311 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647708 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:43:46.659311 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647712 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:43:46.659311 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647715 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:43:46.659311 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:46.647717 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:43:46.659311 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.647773 2576 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:43:46.659311 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.650163 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:43:46.659311 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.651073 2576 server.go:1014] "Starting client certificate rotation" Oct 30 21:43:46.659311 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.651173 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:43:46.659311 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.651199 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:43:46.659311 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.653205 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:43:46.659311 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.654747 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:43:46.667450 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.667433 2576 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:43:46.671090 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.671075 2576 log.go:25] "Validated CRI v1 image API" Oct 30 21:43:46.671861 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.671839 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:43:46.676319 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.676295 2576 fs.go:135] Filesystem UUIDs: map[74dfd489-63be-45b2-9267-ff925c629e5c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d48cc798-0955-4ba8-802c-943780d36880:/dev/nvme0n1p4] Oct 30 21:43:46.676381 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.676317 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:43:46.682757 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.682647 2576 manager.go:217] Machine: {Timestamp:2026-10-30 21:43:46.680208748 +0000 UTC m=+0.324787720 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103202 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23808bec0d619ad363015ef799f5d5 SystemUUID:ec23808b-ec0d-619a-d363-015ef799f5d5 BootID:18424bbf-1c5f-42fc-bc34-9ff446645789 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4b:9f:f8:63:13 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4b:9f:f8:63:13 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:27:19:f0:bb:5e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:43:46.682757 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.682753 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:43:46.682859 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.682818 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:43:46.685975 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.685954 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:43:46.686095 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.685978 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-193.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:43:46.686137 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.686097 2576 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:43:46.686137 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.686105 2576 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:43:46.686137 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.686120 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:43:46.686137 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.686134 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:43:46.687286 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.687277 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:43:46.691685 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.691673 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:43:46.691799 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.691790 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:43:46.692658 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.692648 2576 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:43:46.692694 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.692662 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:43:46.692694 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.692672 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:43:46.692694 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.692681 2576 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:43:46.692694 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.692689 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:43:46.692974 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.692962 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:43:46.695510 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.695485 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:43:46.702639 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.702627 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:43:46.703601 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.703589 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:43:46.703654 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.703636 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:43:46.703807 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.703797 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:43:46.703834 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.703812 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:43:46.703834 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.703818 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:43:46.703834 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.703824 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:43:46.703834 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.703829 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:43:46.703834 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.703835 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:43:46.704636 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.704628 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:43:46.704666 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.704637 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:43:46.704666 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.704644 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:43:46.704666 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.704650 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:43:46.704666 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.704658 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:43:46.704666 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.704667 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:43:46.705553 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.705545 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:43:46.705553 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.705553 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:43:46.709256 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.709243 2576 server.go:1263] "Started kubelet" Oct 30 21:43:46.709327 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.709302 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:43:46.709436 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.709349 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:43:46.709494 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.709466 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:43:46.709912 ip-10-0-131-193 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:43:46.711803 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.711789 2576 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:43:46.715435 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.715407 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:43:46.716105 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:46.716078 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:43:46.722019 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.721999 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:43:46.722586 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.722568 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:43:46.722753 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:46.722701 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-193.us-west-1.compute.internal\" not found" Oct 30 21:43:46.723353 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.723335 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:43:46.723445 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.723338 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:43:46.723445 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.723373 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:43:46.723445 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.723405 2576 factory.go:55] Registering systemd factory Oct 30 21:43:46.723445 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.723421 2576 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:43:46.723648 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.723483 2576 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:43:46.723648 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.723490 2576 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:43:46.723722 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.723685 2576 factory.go:153] Registering CRI-O factory Oct 30 21:43:46.723722 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.723701 2576 factory.go:223] Registration of the crio container factory successfully Oct 30 21:43:46.723811 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.723745 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:43:46.723811 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.723758 2576 factory.go:103] Registering Raw factory Oct 30 21:43:46.723811 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.723767 2576 manager.go:1199] Started watching for new ooms in manager Oct 30 21:43:46.725274 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.725258 2576 manager.go:319] Starting recovery of all containers Oct 30 21:43:46.731530 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.731478 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:43:46.734111 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.734095 2576 manager.go:324] Recovery completed Oct 30 21:43:46.737733 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.737718 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:43:46.739239 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.739224 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:43:46.739297 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.739256 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:43:46.739297 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.739267 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:43:46.739735 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.739720 2576 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:43:46.739735 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.739728 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:43:46.739735 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.739732 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:43:46.739920 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.739748 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:43:46.740378 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.740363 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:43:46.740449 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.740393 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:43:46.740449 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.740411 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:43:46.741779 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.741760 2576 policy_none.go:50] "Start" Oct 30 21:43:46.741779 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.741775 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:43:46.741860 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.741785 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:43:46.743686 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.743675 2576 policy_none.go:44] "Start" Oct 30 21:43:46.802313 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.781863 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:43:46.802313 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.785236 2576 manager.go:341] "Starting Device Plugin manager" Oct 30 21:43:46.802313 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:46.785259 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:43:46.802313 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.785267 2576 server.go:87] "Starting device plugin registration server" Oct 30 21:43:46.802313 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.785449 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:43:46.802313 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.785458 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:43:46.802313 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.785558 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:43:46.802313 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.785657 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:43:46.802313 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.785665 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:43:46.802313 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:46.786176 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:43:46.802313 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:46.786220 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-193.us-west-1.compute.internal\" not found" Oct 30 21:43:46.802313 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.795171 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:43:46.802313 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.795760 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:43:46.802313 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.795792 2576 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:43:46.802313 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:46.795832 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:43:46.857261 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:46.856162 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-193.us-west-1.compute.internal.18e36cbaec62fe95 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-193.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-193.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:43:46.709216917 +0000 UTC m=+0.353795884,LastTimestamp:2026-10-30 21:43:46.709216917 +0000 UTC m=+0.353795884,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-193.us-west-1.compute.internal,}" Oct 30 21:43:46.885787 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.885745 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:43:46.886503 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.886484 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:43:46.886568 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.886517 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:43:46.886568 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.886528 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:43:46.886568 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.886549 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:46.896432 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.896408 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal"] Oct 30 21:43:46.896496 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.896463 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:43:46.897120 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.897108 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:43:46.897171 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.897132 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:43:46.897171 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.897142 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:43:46.899302 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.899290 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:43:46.899435 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.899422 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:46.899478 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.899450 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:43:46.899989 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.899971 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:43:46.900096 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.899974 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:43:46.900096 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.900031 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:43:46.900096 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.900050 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:43:46.900096 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.900002 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:43:46.900096 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.900083 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:43:46.902166 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.902151 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:46.902228 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.902174 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:43:46.902840 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.902824 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:43:46.902840 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.902851 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:43:46.902969 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.902866 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:43:46.904410 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.904392 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-193.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:43:46.904518 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:46.904500 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-193.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:43:46.923032 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:46.923012 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-193.us-west-1.compute.internal\" not found" node="ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:46.924899 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:46.924882 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-193.us-west-1.compute.internal\" not found" node="ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:46.925133 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.925114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/de7a233ef9a012d605524a0af5a13d0b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal\" (UID: \"de7a233ef9a012d605524a0af5a13d0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:46.925209 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.925139 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de7a233ef9a012d605524a0af5a13d0b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal\" (UID: \"de7a233ef9a012d605524a0af5a13d0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:46.925209 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.925160 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b76742bc66466f5dd9bc77bf5370e40-config\") pod \"kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal\" (UID: \"8b76742bc66466f5dd9bc77bf5370e40\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:46.934585 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.934565 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:43:46.934664 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:46.934627 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-57xcj" Oct 30 21:43:46.954485 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:46.954402 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-193.us-west-1.compute.internal.18e36cbaee2d1c54 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-193.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-193.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-193.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:43:46.73924002 +0000 UTC m=+0.383818993,LastTimestamp:2026-10-30 21:43:46.73924002 +0000 UTC m=+0.383818993,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-193.us-west-1.compute.internal,}" Oct 30 21:43:46.954606 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:46.954492 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 21:43:46.954606 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:46.954525 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 21:43:47.019489 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.019472 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:43:47.020292 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:47.020277 2576 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-193.us-west-1.compute.internal\" not found" node="ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:47.021425 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.021412 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:43:47.025358 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.025343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/de7a233ef9a012d605524a0af5a13d0b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal\" (UID: \"de7a233ef9a012d605524a0af5a13d0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:47.025423 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.025365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de7a233ef9a012d605524a0af5a13d0b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal\" (UID: \"de7a233ef9a012d605524a0af5a13d0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:47.025423 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.025381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b76742bc66466f5dd9bc77bf5370e40-config\") pod \"kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal\" (UID: \"8b76742bc66466f5dd9bc77bf5370e40\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:47.025517 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.025433 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b76742bc66466f5dd9bc77bf5370e40-config\") pod \"kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal\" (UID: \"8b76742bc66466f5dd9bc77bf5370e40\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:47.025517 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.025458 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de7a233ef9a012d605524a0af5a13d0b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal\" (UID: \"de7a233ef9a012d605524a0af5a13d0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:47.025517 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.025468 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/de7a233ef9a012d605524a0af5a13d0b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal\" (UID: \"de7a233ef9a012d605524a0af5a13d0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:47.210027 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:47.209980 2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-193.us-west-1.compute.internal\" not found" node="ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:47.221108 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.221087 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:43:47.222021 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.222003 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:43:47.222111 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.222036 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:43:47.222111 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.222052 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:43:47.222111 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.222080 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:47.228566 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.228551 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:47.230740 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.230725 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:47.280327 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.280305 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:47.323061 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.323038 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:47.394216 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.394192 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:43:47.395295 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.395282 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:43:47.455098 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.455075 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:43:47.651256 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.651203 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:43:47.651708 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.651368 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:43:47.651708 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:47.651345 2576 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a9858833090b34fb5b824294711bfe6c-e75a63cf43fa0c23.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-131-193.us-west-1.compute.internal.18e36cbaee2d710b\": read tcp 10.0.131.193:50392->3.149.172.21:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-131-193.us-west-1.compute.internal.18e36cbaee2d710b default 5034 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-193.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-193.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-193.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:43:46 +0000 UTC,LastTimestamp:2026-10-30 21:43:46.897136286 +0000 UTC m=+0.541715252,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-193.us-west-1.compute.internal,}" Oct 30 21:43:47.693215 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.693184 2576 apiserver.go:52] "Watching apiserver" Oct 30 21:43:47.722407 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.722384 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:43:47.941114 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.941049 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:38:47 +0000 UTC" deadline="2028-08-16 16:16:00.628112378 +0000 UTC" Oct 30 21:43:47.941114 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.941077 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15738h32m12.687038012s" Oct 30 21:43:47.966893 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.966862 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:43:47.967234 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.967209 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-wh7rd","openshift-cluster-node-tuning-operator/tuned-89cpv","openshift-multus/multus-additional-cni-plugins-n6b4f","openshift-network-operator/iptables-alerter-vkl9r","openshift-ovn-kubernetes/ovnkube-node-w6p4f","kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal","openshift-image-registry/node-ca-45jdt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal","kube-system/global-pull-secret-syncer-8mr69","kube-system/konnectivity-agent-lpx2r","openshift-dns/node-resolver-vx4gh","openshift-network-diagnostics/network-check-target-tcckj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl","openshift-multus/multus-b5fb6"] Oct 30 21:43:47.969728 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.969706 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:43:47.969815 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:47.969777 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wh7rd" podUID="c94b6070-d303-4c76-8895-73cd37a2f985" Oct 30 21:43:47.971812 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.971796 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:47.973981 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.973953 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:47.976113 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.976095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vkl9r" Oct 30 21:43:47.978454 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.978437 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:47.980703 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.980683 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-45jdt" Oct 30 21:43:47.982925 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.982910 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:47.982993 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:47.982963 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8mr69" podUID="f73329a3-7290-44a1-a72e-1fe72c0d0da5" Oct 30 21:43:47.985065 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.985044 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lpx2r" Oct 30 21:43:47.987457 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.987436 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vx4gh" Oct 30 21:43:47.989809 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.989786 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:43:47.989918 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:47.989842 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcckj" podUID="3ee9e6d0-f122-4ff3-aa72-b597339928ac" Oct 30 21:43:47.992261 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.992242 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:47.994468 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:47.994452 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.024216 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.024195 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:43:48.027174 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.025302 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:43:48.027174 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.025325 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:43:48.027174 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.025338 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s7wrh\"" Oct 30 21:43:48.027174 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.026548 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:43:48.027174 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.026851 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:43:48.027174 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.026938 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:43:48.027174 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.027023 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9vv2h\"" Oct 30 21:43:48.027174 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.027101 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:43:48.027649 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.027593 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:43:48.028753 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.028738 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-22kpt\"" Oct 30 21:43:48.028806 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.028780 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:43:48.028964 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.028950 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:43:48.028964 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.028959 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:43:48.031081 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031065 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-cni-bin\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.031123 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031089 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-registration-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.031123 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-etc-selinux\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.031123 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535-iptables-alerter-script\") pod \"iptables-alerter-vkl9r\" (UID: \"4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535\") " pod="openshift-network-operator/iptables-alerter-vkl9r" Oct 30 21:43:48.031210 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031134 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a899495-aa25-4538-8eef-5640503b466a-cni-binary-copy\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.031210 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031148 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzbrt\" (UniqueName: \"kubernetes.io/projected/c94b6070-d303-4c76-8895-73cd37a2f985-kube-api-access-bzbrt\") pod \"network-metrics-daemon-wh7rd\" (UID: \"c94b6070-d303-4c76-8895-73cd37a2f985\") " pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:43:48.031210 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031200 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-kubelet\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.031339 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031235 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08685d97-2f18-4abe-a7b8-25466aee77cb-ovnkube-script-lib\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.031339 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031265 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f73329a3-7290-44a1-a72e-1fe72c0d0da5-kubelet-config\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:48.031339 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/08df8d3b-f392-438a-81d5-3ee04ee2cd8b-konnectivity-ca\") pod \"konnectivity-agent-lpx2r\" (UID: \"08df8d3b-f392-438a-81d5-3ee04ee2cd8b\") " pod="kube-system/konnectivity-agent-lpx2r" Oct 30 21:43:48.031339 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee354605-03c9-4c69-a937-5f94440dc0ae-serviceca\") pod \"node-ca-45jdt\" (UID: \"ee354605-03c9-4c69-a937-5f94440dc0ae\") " pod="openshift-image-registry/node-ca-45jdt" Oct 30 21:43:48.031506 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs\") pod \"network-metrics-daemon-wh7rd\" (UID: \"c94b6070-d303-4c76-8895-73cd37a2f985\") " pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:43:48.031506 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031364 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-modprobe-d\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.031506 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031378 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.031506 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031383 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:43:48.031506 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.031506 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031416 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hkrw6\"" Oct 30 21:43:48.031506 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031436 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:43:48.031506 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031451 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:43:48.031506 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031470 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:43:48.031506 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031488 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:43:48.031927 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031646 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:43:48.031927 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031670 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-host\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.031927 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031693 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-run-openvswitch\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.031927 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-node-log\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.031927 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031738 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.031927 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031780 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.031927 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-etc-kubernetes\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.031927 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-sys\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.031927 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-var-lib-kubelet\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.031927 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08685d97-2f18-4abe-a7b8-25466aee77cb-ovn-node-metrics-cert\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.031927 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031896 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-cnibin\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.031927 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-sysctl-conf\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.032385 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-run-ovn\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.032385 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.031994 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nt72\" (UniqueName: \"kubernetes.io/projected/ee354605-03c9-4c69-a937-5f94440dc0ae-kube-api-access-7nt72\") pod \"node-ca-45jdt\" (UID: \"ee354605-03c9-4c69-a937-5f94440dc0ae\") " pod="openshift-image-registry/node-ca-45jdt" Oct 30 21:43:48.032385 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032033 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9qk6\" (UniqueName: \"kubernetes.io/projected/827f9fb3-0eb3-4f8f-96f2-72df80c024d8-kube-api-access-v9qk6\") pod \"node-resolver-vx4gh\" (UID: \"827f9fb3-0eb3-4f8f-96f2-72df80c024d8\") " pod="openshift-dns/node-resolver-vx4gh" Oct 30 21:43:48.032385 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a899495-aa25-4538-8eef-5640503b466a-multus-daemon-config\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.032385 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-system-cni-dir\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.032385 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032121 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-etc-openvswitch\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.032385 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032155 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74fe5a17-9648-48a9-8d1b-338990dac55b-tmp\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.032385 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032189 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.032385 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032213 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-cni-netd\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.032385 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032255 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08685d97-2f18-4abe-a7b8-25466aee77cb-ovnkube-config\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.032385 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032304 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/08df8d3b-f392-438a-81d5-3ee04ee2cd8b-agent-certs\") pod \"konnectivity-agent-lpx2r\" (UID: \"08df8d3b-f392-438a-81d5-3ee04ee2cd8b\") " pod="kube-system/konnectivity-agent-lpx2r" Oct 30 21:43:48.032385 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032340 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-device-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.032385 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032371 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-run\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.032850 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46rpn\" (UniqueName: \"kubernetes.io/projected/74fe5a17-9648-48a9-8d1b-338990dac55b-kube-api-access-46rpn\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.032850 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-socket-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.032850 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.032850 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-multus-cni-dir\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.032850 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-var-lib-cni-bin\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.032850 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032565 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08685d97-2f18-4abe-a7b8-25466aee77cb-env-overrides\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.032850 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbn2h\" (UniqueName: \"kubernetes.io/projected/08685d97-2f18-4abe-a7b8-25466aee77cb-kube-api-access-hbn2h\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.032850 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032610 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f73329a3-7290-44a1-a72e-1fe72c0d0da5-dbus\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:48.032850 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032653 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-sys-fs\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.032850 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032683 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.032850 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032734 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535-host-slash\") pod \"iptables-alerter-vkl9r\" (UID: \"4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535\") " pod="openshift-network-operator/iptables-alerter-vkl9r" Oct 30 21:43:48.032850 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032783 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-cnibin\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.032850 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-var-lib-cni-multus\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.032850 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032847 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-sysconfig\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.033231 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-systemd-units\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.033231 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032912 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjtfp\" (UniqueName: \"kubernetes.io/projected/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535-kube-api-access-hjtfp\") pod \"iptables-alerter-vkl9r\" (UID: \"4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535\") " pod="openshift-network-operator/iptables-alerter-vkl9r" Oct 30 21:43:48.033231 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032952 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-multus-socket-dir-parent\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.033231 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.032987 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-sysctl-d\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.033231 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033017 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-lib-modules\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.033231 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033055 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-slash\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.033231 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033084 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-log-socket\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.033231 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827f9fb3-0eb3-4f8f-96f2-72df80c024d8-tmp-dir\") pod \"node-resolver-vx4gh\" (UID: \"827f9fb3-0eb3-4f8f-96f2-72df80c024d8\") " pod="openshift-dns/node-resolver-vx4gh" Oct 30 21:43:48.033231 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033154 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-run-netns\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.033231 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8l9c\" (UniqueName: \"kubernetes.io/projected/2a899495-aa25-4538-8eef-5640503b466a-kube-api-access-p8l9c\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.033231 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033201 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-systemd\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.033231 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033229 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-run-netns\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.033573 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033274 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee354605-03c9-4c69-a937-5f94440dc0ae-host\") pod \"node-ca-45jdt\" (UID: \"ee354605-03c9-4c69-a937-5f94440dc0ae\") " pod="openshift-image-registry/node-ca-45jdt" Oct 30 21:43:48.033573 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-multus-conf-dir\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.033573 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-run-k8s-cni-cncf-io\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.033573 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033340 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-hostroot\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.033573 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033377 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-kubernetes\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.033573 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-tuned\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.033573 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033414 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/827f9fb3-0eb3-4f8f-96f2-72df80c024d8-hosts-file\") pod \"node-resolver-vx4gh\" (UID: \"827f9fb3-0eb3-4f8f-96f2-72df80c024d8\") " pod="openshift-dns/node-resolver-vx4gh" Oct 30 21:43:48.033573 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cbbb\" (UniqueName: \"kubernetes.io/projected/e1668e0e-55e0-493c-a735-1ff0da66eda3-kube-api-access-5cbbb\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.033573 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-run-systemd\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.033573 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-var-lib-openvswitch\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.033573 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:48.033573 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033539 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-run-multus-certs\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.033573 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033568 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cgbv\" (UniqueName: \"kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv\") pod \"network-check-target-tcckj\" (UID: \"3ee9e6d0-f122-4ff3-aa72-b597339928ac\") " pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:43:48.034038 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wfjl\" (UniqueName: \"kubernetes.io/projected/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-kube-api-access-4wfjl\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.034038 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033599 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-system-cni-dir\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.034038 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.033665 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-var-lib-kubelet\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.071122 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.071103 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6hfjp\"" Oct 30 21:43:48.071265 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.071252 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:43:48.071265 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.071258 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:43:48.071369 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.071284 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:43:48.072813 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.072796 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:43:48.072813 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.072795 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gmj65\"" Oct 30 21:43:48.072929 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.072820 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:43:48.074664 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.074566 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:43:48.074664 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.074647 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v5cb9" Oct 30 21:43:48.126192 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.126176 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:43:48.126255 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.126177 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:43:48.126465 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.126452 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:43:48.126504 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.126480 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:43:48.126732 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.126718 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:43:48.127290 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.127277 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:43:48.127502 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.127490 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5pc99\"" Oct 30 21:43:48.127558 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.127494 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vv25x\"" Oct 30 21:43:48.127558 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.127524 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fqpcl\"" Oct 30 21:43:48.134527 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.134513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-etc-selinux\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.134583 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.134538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535-iptables-alerter-script\") pod \"iptables-alerter-vkl9r\" (UID: \"4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535\") " pod="openshift-network-operator/iptables-alerter-vkl9r" Oct 30 21:43:48.134583 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.134556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a899495-aa25-4538-8eef-5640503b466a-cni-binary-copy\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.134583 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.134570 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzbrt\" (UniqueName: \"kubernetes.io/projected/c94b6070-d303-4c76-8895-73cd37a2f985-kube-api-access-bzbrt\") pod \"network-metrics-daemon-wh7rd\" (UID: \"c94b6070-d303-4c76-8895-73cd37a2f985\") " pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:43:48.134743 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.134584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-kubelet\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.134743 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.134598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08685d97-2f18-4abe-a7b8-25466aee77cb-ovnkube-script-lib\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.134743 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.134627 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f73329a3-7290-44a1-a72e-1fe72c0d0da5-kubelet-config\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:48.134743 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.134685 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/08df8d3b-f392-438a-81d5-3ee04ee2cd8b-konnectivity-ca\") pod \"konnectivity-agent-lpx2r\" (UID: \"08df8d3b-f392-438a-81d5-3ee04ee2cd8b\") " pod="kube-system/konnectivity-agent-lpx2r" Oct 30 21:43:48.134743 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.134703 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f73329a3-7290-44a1-a72e-1fe72c0d0da5-kubelet-config\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:48.134743 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.134713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee354605-03c9-4c69-a937-5f94440dc0ae-serviceca\") pod \"node-ca-45jdt\" (UID: \"ee354605-03c9-4c69-a937-5f94440dc0ae\") " pod="openshift-image-registry/node-ca-45jdt" Oct 30 21:43:48.134743 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.134736 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-etc-selinux\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.135055 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.134740 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs\") pod \"network-metrics-daemon-wh7rd\" (UID: \"c94b6070-d303-4c76-8895-73cd37a2f985\") " pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:43:48.135055 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:48.134810 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:43:48.135055 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:48.134886 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs podName:c94b6070-d303-4c76-8895-73cd37a2f985 nodeName:}" failed. No retries permitted until 2026-10-30 21:43:48.634847385 +0000 UTC m=+2.279426353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs") pod "network-metrics-daemon-wh7rd" (UID: "c94b6070-d303-4c76-8895-73cd37a2f985") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:43:48.135055 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.134928 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-kubelet\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.135242 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135094 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-modprobe-d\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.135242 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135219 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535-iptables-alerter-script\") pod \"iptables-alerter-vkl9r\" (UID: \"4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535\") " pod="openshift-network-operator/iptables-alerter-vkl9r" Oct 30 21:43:48.135340 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135276 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/08df8d3b-f392-438a-81d5-3ee04ee2cd8b-konnectivity-ca\") pod \"konnectivity-agent-lpx2r\" (UID: \"08df8d3b-f392-438a-81d5-3ee04ee2cd8b\") " pod="kube-system/konnectivity-agent-lpx2r" Oct 30 21:43:48.135340 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135287 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee354605-03c9-4c69-a937-5f94440dc0ae-serviceca\") pod \"node-ca-45jdt\" (UID: \"ee354605-03c9-4c69-a937-5f94440dc0ae\") " pod="openshift-image-registry/node-ca-45jdt" Oct 30 21:43:48.135340 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.134789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-modprobe-d\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.135455 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135344 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a899495-aa25-4538-8eef-5640503b466a-cni-binary-copy\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.135455 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.135455 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135390 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.135455 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-host\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.135455 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135416 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08685d97-2f18-4abe-a7b8-25466aee77cb-ovnkube-script-lib\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.135455 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-run-openvswitch\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.135455 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135438 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.135799 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135462 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-run-openvswitch\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.135799 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135490 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-host\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.135799 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135523 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-node-log\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.135799 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135530 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.135799 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.135799 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135566 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-node-log\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.135799 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135571 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.135799 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-etc-kubernetes\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.135799 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135629 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-etc-kubernetes\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.135799 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135632 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.135799 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135646 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-sys\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.135799 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135683 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-sys\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.135799 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-var-lib-kubelet\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.135799 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08685d97-2f18-4abe-a7b8-25466aee77cb-ovn-node-metrics-cert\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.135799 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-cnibin\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.135799 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-sysctl-conf\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.135799 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135732 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-var-lib-kubelet\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.136604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-run-ovn\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.136604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135817 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-cnibin\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.136604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nt72\" (UniqueName: \"kubernetes.io/projected/ee354605-03c9-4c69-a937-5f94440dc0ae-kube-api-access-7nt72\") pod \"node-ca-45jdt\" (UID: \"ee354605-03c9-4c69-a937-5f94440dc0ae\") " pod="openshift-image-registry/node-ca-45jdt" Oct 30 21:43:48.136604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135845 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-run-ovn\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.136604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135845 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9qk6\" (UniqueName: \"kubernetes.io/projected/827f9fb3-0eb3-4f8f-96f2-72df80c024d8-kube-api-access-v9qk6\") pod \"node-resolver-vx4gh\" (UID: \"827f9fb3-0eb3-4f8f-96f2-72df80c024d8\") " pod="openshift-dns/node-resolver-vx4gh" Oct 30 21:43:48.136604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a899495-aa25-4538-8eef-5640503b466a-multus-daemon-config\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.136604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135892 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-system-cni-dir\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.136604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-etc-openvswitch\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.136604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74fe5a17-9648-48a9-8d1b-338990dac55b-tmp\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.136604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.136604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135973 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-etc-openvswitch\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.136604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136002 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.136604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-cni-netd\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.136604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136041 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08685d97-2f18-4abe-a7b8-25466aee77cb-ovnkube-config\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.136604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136047 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-system-cni-dir\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.136604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136015 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:43:48.136604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136066 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.136604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/08df8d3b-f392-438a-81d5-3ee04ee2cd8b-agent-certs\") pod \"konnectivity-agent-lpx2r\" (UID: \"08df8d3b-f392-438a-81d5-3ee04ee2cd8b\") " pod="kube-system/konnectivity-agent-lpx2r" Oct 30 21:43:48.137596 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136082 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-cni-netd\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.137596 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.135873 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-sysctl-conf\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.137596 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136119 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-device-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.137596 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136203 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-device-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.137596 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-run\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.137596 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136256 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46rpn\" (UniqueName: \"kubernetes.io/projected/74fe5a17-9648-48a9-8d1b-338990dac55b-kube-api-access-46rpn\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.137596 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-socket-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.137596 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.137596 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-multus-cni-dir\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.137596 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136366 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-var-lib-cni-bin\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.137596 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136371 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a899495-aa25-4538-8eef-5640503b466a-multus-daemon-config\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.137596 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136383 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-run\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.137596 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08685d97-2f18-4abe-a7b8-25466aee77cb-env-overrides\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.137596 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbn2h\" (UniqueName: \"kubernetes.io/projected/08685d97-2f18-4abe-a7b8-25466aee77cb-kube-api-access-hbn2h\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.137596 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136445 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f73329a3-7290-44a1-a72e-1fe72c0d0da5-dbus\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:48.137596 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136468 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-sys-fs\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.137596 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136488 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-multus-cni-dir\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.138376 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136495 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.138376 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535-host-slash\") pod \"iptables-alerter-vkl9r\" (UID: \"4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535\") " pod="openshift-network-operator/iptables-alerter-vkl9r" Oct 30 21:43:48.138376 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-cnibin\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.138376 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136564 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08685d97-2f18-4abe-a7b8-25466aee77cb-ovnkube-config\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.138376 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-var-lib-cni-multus\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.138376 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-sysconfig\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.138376 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136656 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-systemd-units\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.138376 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjtfp\" (UniqueName: \"kubernetes.io/projected/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535-kube-api-access-hjtfp\") pod \"iptables-alerter-vkl9r\" (UID: \"4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535\") " pod="openshift-network-operator/iptables-alerter-vkl9r" Oct 30 21:43:48.138376 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136696 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-multus-socket-dir-parent\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.138376 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-sysctl-d\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.138376 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-lib-modules\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.138376 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-slash\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.138376 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-log-socket\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.138376 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136793 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827f9fb3-0eb3-4f8f-96f2-72df80c024d8-tmp-dir\") pod \"node-resolver-vx4gh\" (UID: \"827f9fb3-0eb3-4f8f-96f2-72df80c024d8\") " pod="openshift-dns/node-resolver-vx4gh" Oct 30 21:43:48.138376 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136811 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-run-netns\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.138376 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8l9c\" (UniqueName: \"kubernetes.io/projected/2a899495-aa25-4538-8eef-5640503b466a-kube-api-access-p8l9c\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.138376 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136844 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.139147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-systemd\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.139147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-run-netns\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.139147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136899 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee354605-03c9-4c69-a937-5f94440dc0ae-host\") pod \"node-ca-45jdt\" (UID: \"ee354605-03c9-4c69-a937-5f94440dc0ae\") " pod="openshift-image-registry/node-ca-45jdt" Oct 30 21:43:48.139147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136902 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-var-lib-cni-multus\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.139147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136920 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.139147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136971 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-multus-socket-dir-parent\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.139147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.136993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-multus-conf-dir\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.139147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137016 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-var-lib-cni-bin\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.139147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137057 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-run-k8s-cni-cncf-io\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.139147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137083 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-sysconfig\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.139147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137125 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-systemd-units\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.139147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137181 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535-host-slash\") pod \"iptables-alerter-vkl9r\" (UID: \"4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535\") " pod="openshift-network-operator/iptables-alerter-vkl9r" Oct 30 21:43:48.139147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137225 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-multus-conf-dir\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.139147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137246 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-sysctl-d\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.139147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137325 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-socket-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.139147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137367 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-cnibin\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.139147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137376 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f73329a3-7290-44a1-a72e-1fe72c0d0da5-dbus\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:48.139147 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137022 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-run-k8s-cni-cncf-io\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.139892 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137410 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee354605-03c9-4c69-a937-5f94440dc0ae-host\") pod \"node-ca-45jdt\" (UID: \"ee354605-03c9-4c69-a937-5f94440dc0ae\") " pod="openshift-image-registry/node-ca-45jdt" Oct 30 21:43:48.139892 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-hostroot\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.139892 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137455 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-sys-fs\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.139892 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-kubernetes\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.139892 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137486 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-tuned\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.139892 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137495 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-hostroot\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.139892 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/827f9fb3-0eb3-4f8f-96f2-72df80c024d8-hosts-file\") pod \"node-resolver-vx4gh\" (UID: \"827f9fb3-0eb3-4f8f-96f2-72df80c024d8\") " pod="openshift-dns/node-resolver-vx4gh" Oct 30 21:43:48.139892 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cbbb\" (UniqueName: \"kubernetes.io/projected/e1668e0e-55e0-493c-a735-1ff0da66eda3-kube-api-access-5cbbb\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.139892 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137540 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-kubernetes\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.139892 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137550 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-slash\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.139892 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137562 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-run-systemd\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.139892 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-var-lib-openvswitch\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.139892 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137582 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-run-netns\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.139892 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:48.139892 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137662 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-run-multus-certs\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.139892 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137668 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-log-socket\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.139892 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cgbv\" (UniqueName: \"kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv\") pod \"network-check-target-tcckj\" (UID: \"3ee9e6d0-f122-4ff3-aa72-b597339928ac\") " pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:43:48.139892 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wfjl\" (UniqueName: \"kubernetes.io/projected/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-kube-api-access-4wfjl\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.140387 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-system-cni-dir\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.140387 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137741 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08685d97-2f18-4abe-a7b8-25466aee77cb-env-overrides\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.140387 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137763 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-var-lib-kubelet\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.140387 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-cni-bin\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.140387 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137792 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-var-lib-openvswitch\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.140387 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137807 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-systemd\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.140387 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137859 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-run-multus-certs\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.140387 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137859 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/827f9fb3-0eb3-4f8f-96f2-72df80c024d8-hosts-file\") pod \"node-resolver-vx4gh\" (UID: \"827f9fb3-0eb3-4f8f-96f2-72df80c024d8\") " pod="openshift-dns/node-resolver-vx4gh" Oct 30 21:43:48.140387 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137511 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-lib-modules\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.140387 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137940 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827f9fb3-0eb3-4f8f-96f2-72df80c024d8-tmp-dir\") pod \"node-resolver-vx4gh\" (UID: \"827f9fb3-0eb3-4f8f-96f2-72df80c024d8\") " pod="openshift-dns/node-resolver-vx4gh" Oct 30 21:43:48.140387 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.137809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-registration-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.140387 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.138117 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-run-netns\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.140387 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.138140 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-var-lib-kubelet\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.140387 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.138164 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-system-cni-dir\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.140387 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.138174 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-run-systemd\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.140387 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.138196 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-cni-bin\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.140387 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.138213 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-registration-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.140387 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:48.138293 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:43:48.140876 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:48.138342 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret podName:f73329a3-7290-44a1-a72e-1fe72c0d0da5 nodeName:}" failed. No retries permitted until 2026-10-30 21:43:48.638328341 +0000 UTC m=+2.282907308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret") pod "global-pull-secret-syncer-8mr69" (UID: "f73329a3-7290-44a1-a72e-1fe72c0d0da5") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:43:48.140876 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.139785 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08685d97-2f18-4abe-a7b8-25466aee77cb-ovn-node-metrics-cert\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.140876 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.139875 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/08df8d3b-f392-438a-81d5-3ee04ee2cd8b-agent-certs\") pod \"konnectivity-agent-lpx2r\" (UID: \"08df8d3b-f392-438a-81d5-3ee04ee2cd8b\") " pod="kube-system/konnectivity-agent-lpx2r" Oct 30 21:43:48.140876 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.139915 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74fe5a17-9648-48a9-8d1b-338990dac55b-tmp\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.140876 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.140014 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-tuned\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.196198 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.196153 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzbrt\" (UniqueName: \"kubernetes.io/projected/c94b6070-d303-4c76-8895-73cd37a2f985-kube-api-access-bzbrt\") pod \"network-metrics-daemon-wh7rd\" (UID: \"c94b6070-d303-4c76-8895-73cd37a2f985\") " pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:43:48.196917 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.196890 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9qk6\" (UniqueName: \"kubernetes.io/projected/827f9fb3-0eb3-4f8f-96f2-72df80c024d8-kube-api-access-v9qk6\") pod \"node-resolver-vx4gh\" (UID: \"827f9fb3-0eb3-4f8f-96f2-72df80c024d8\") " pod="openshift-dns/node-resolver-vx4gh" Oct 30 21:43:48.197172 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.197153 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nt72\" (UniqueName: \"kubernetes.io/projected/ee354605-03c9-4c69-a937-5f94440dc0ae-kube-api-access-7nt72\") pod \"node-ca-45jdt\" (UID: \"ee354605-03c9-4c69-a937-5f94440dc0ae\") " pod="openshift-image-registry/node-ca-45jdt" Oct 30 21:43:48.202945 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.202914 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbn2h\" (UniqueName: \"kubernetes.io/projected/08685d97-2f18-4abe-a7b8-25466aee77cb-kube-api-access-hbn2h\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.205549 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.205522 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46rpn\" (UniqueName: \"kubernetes.io/projected/74fe5a17-9648-48a9-8d1b-338990dac55b-kube-api-access-46rpn\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.205766 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.205748 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjtfp\" (UniqueName: \"kubernetes.io/projected/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535-kube-api-access-hjtfp\") pod \"iptables-alerter-vkl9r\" (UID: \"4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535\") " pod="openshift-network-operator/iptables-alerter-vkl9r" Oct 30 21:43:48.206025 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:48.206007 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:43:48.206086 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:48.206030 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:43:48.206086 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:48.206043 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5cgbv for pod openshift-network-diagnostics/network-check-target-tcckj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:43:48.206181 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:48.206102 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv podName:3ee9e6d0-f122-4ff3-aa72-b597339928ac nodeName:}" failed. No retries permitted until 2026-10-30 21:43:48.706087211 +0000 UTC m=+2.350666181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5cgbv" (UniqueName: "kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv") pod "network-check-target-tcckj" (UID: "3ee9e6d0-f122-4ff3-aa72-b597339928ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:43:48.207392 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.207363 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wfjl\" (UniqueName: \"kubernetes.io/projected/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-kube-api-access-4wfjl\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.208542 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.208520 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8l9c\" (UniqueName: \"kubernetes.io/projected/2a899495-aa25-4538-8eef-5640503b466a-kube-api-access-p8l9c\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.209411 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.209387 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cbbb\" (UniqueName: \"kubernetes.io/projected/e1668e0e-55e0-493c-a735-1ff0da66eda3-kube-api-access-5cbbb\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.233531 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.233514 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:43:48.314946 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.314908 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:43:48.352972 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.352949 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vkl9r" Oct 30 21:43:48.354084 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.354066 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:43:48.387043 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.387026 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:43:48.392155 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.392136 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-45jdt" Oct 30 21:43:48.392252 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.392195 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:43:48.413698 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.413676 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lpx2r" Oct 30 21:43:48.420283 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.420269 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vx4gh" Oct 30 21:43:48.434253 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:48.434221 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827f9fb3_0eb3_4f8f_96f2_72df80c024d8.slice/crio-afa919c15cf04a00e9c88be0ab38c60002f01b4d0fa843a67ceeb4f034748f85 WatchSource:0}: Error finding container afa919c15cf04a00e9c88be0ab38c60002f01b4d0fa843a67ceeb4f034748f85: Status 404 returned error can't find the container with id afa919c15cf04a00e9c88be0ab38c60002f01b4d0fa843a67ceeb4f034748f85 Oct 30 21:43:48.434816 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:48.434795 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08df8d3b_f392_438a_81d5_3ee04ee2cd8b.slice/crio-ff888be3d101a97039d7b7ff621081a65a1d5bf86559c62342cc75095a572cfb WatchSource:0}: Error finding container ff888be3d101a97039d7b7ff621081a65a1d5bf86559c62342cc75095a572cfb: Status 404 returned error can't find the container with id ff888be3d101a97039d7b7ff621081a65a1d5bf86559c62342cc75095a572cfb Oct 30 21:43:48.435542 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:48.435521 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b76742bc66466f5dd9bc77bf5370e40.slice/crio-4d1b6a0958405e70a9f74d458070972d7112ad956c090e4ac078281e0474474a WatchSource:0}: Error finding container 4d1b6a0958405e70a9f74d458070972d7112ad956c090e4ac078281e0474474a: Status 404 returned error can't find the container with id 4d1b6a0958405e70a9f74d458070972d7112ad956c090e4ac078281e0474474a Oct 30 21:43:48.436786 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:48.436762 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7a233ef9a012d605524a0af5a13d0b.slice/crio-03516dcf44f10b5c29eaccb99085929e2c26bbe9b1dd584f83a2034331c977fb WatchSource:0}: Error finding container 03516dcf44f10b5c29eaccb99085929e2c26bbe9b1dd584f83a2034331c977fb: Status 404 returned error can't find the container with id 03516dcf44f10b5c29eaccb99085929e2c26bbe9b1dd584f83a2034331c977fb Oct 30 21:43:48.439604 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.439587 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:43:48.447261 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.447227 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:43:48.451986 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.451970 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b5fb6" Oct 30 21:43:48.456249 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:48.456232 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1668e0e_55e0_493c_a735_1ff0da66eda3.slice/crio-7450f996eafba6511069b0b4bd3c37035e90292311482e39304f970f045b75c6 WatchSource:0}: Error finding container 7450f996eafba6511069b0b4bd3c37035e90292311482e39304f970f045b75c6: Status 404 returned error can't find the container with id 7450f996eafba6511069b0b4bd3c37035e90292311482e39304f970f045b75c6 Oct 30 21:43:48.460985 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:48.460966 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a899495_aa25_4538_8eef_5640503b466a.slice/crio-f8d8f5bd04a7a184c288d5eda923787389f21397007a832e3d324e56377077d0 WatchSource:0}: Error finding container f8d8f5bd04a7a184c288d5eda923787389f21397007a832e3d324e56377077d0: Status 404 returned error can't find the container with id f8d8f5bd04a7a184c288d5eda923787389f21397007a832e3d324e56377077d0 Oct 30 21:43:48.640870 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.640839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:48.640950 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.640886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs\") pod \"network-metrics-daemon-wh7rd\" (UID: \"c94b6070-d303-4c76-8895-73cd37a2f985\") " pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:43:48.641009 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:48.640995 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:43:48.641056 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:48.641046 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret podName:f73329a3-7290-44a1-a72e-1fe72c0d0da5 nodeName:}" failed. No retries permitted until 2026-10-30 21:43:49.641032777 +0000 UTC m=+3.285611746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret") pod "global-pull-secret-syncer-8mr69" (UID: "f73329a3-7290-44a1-a72e-1fe72c0d0da5") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:43:48.641099 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:48.640995 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:43:48.641099 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:48.641087 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs podName:c94b6070-d303-4c76-8895-73cd37a2f985 nodeName:}" failed. No retries permitted until 2026-10-30 21:43:49.641075713 +0000 UTC m=+3.285654683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs") pod "network-metrics-daemon-wh7rd" (UID: "c94b6070-d303-4c76-8895-73cd37a2f985") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:43:48.741800 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.741748 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cgbv\" (UniqueName: \"kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv\") pod \"network-check-target-tcckj\" (UID: \"3ee9e6d0-f122-4ff3-aa72-b597339928ac\") " pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:43:48.742307 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:48.741897 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:43:48.742307 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:48.741913 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:43:48.742307 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:48.741922 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5cgbv for pod openshift-network-diagnostics/network-check-target-tcckj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:43:48.742307 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:48.741971 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv podName:3ee9e6d0-f122-4ff3-aa72-b597339928ac nodeName:}" failed. No retries permitted until 2026-10-30 21:43:49.741959236 +0000 UTC m=+3.386538205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5cgbv" (UniqueName: "kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv") pod "network-check-target-tcckj" (UID: "3ee9e6d0-f122-4ff3-aa72-b597339928ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:43:48.799778 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.799742 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b5fb6" event={"ID":"2a899495-aa25-4538-8eef-5640503b466a","Type":"ContainerStarted","Data":"f8d8f5bd04a7a184c288d5eda923787389f21397007a832e3d324e56377077d0"} Oct 30 21:43:48.801366 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.801339 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" event={"ID":"e1668e0e-55e0-493c-a735-1ff0da66eda3","Type":"ContainerStarted","Data":"7450f996eafba6511069b0b4bd3c37035e90292311482e39304f970f045b75c6"} Oct 30 21:43:48.802480 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.802452 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vx4gh" event={"ID":"827f9fb3-0eb3-4f8f-96f2-72df80c024d8","Type":"ContainerStarted","Data":"afa919c15cf04a00e9c88be0ab38c60002f01b4d0fa843a67ceeb4f034748f85"} Oct 30 21:43:48.803695 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.803676 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lpx2r" event={"ID":"08df8d3b-f392-438a-81d5-3ee04ee2cd8b","Type":"ContainerStarted","Data":"ff888be3d101a97039d7b7ff621081a65a1d5bf86559c62342cc75095a572cfb"} Oct 30 21:43:48.804661 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.804641 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" event={"ID":"de7a233ef9a012d605524a0af5a13d0b","Type":"ContainerStarted","Data":"03516dcf44f10b5c29eaccb99085929e2c26bbe9b1dd584f83a2034331c977fb"} Oct 30 21:43:48.805686 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.805662 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal" event={"ID":"8b76742bc66466f5dd9bc77bf5370e40","Type":"ContainerStarted","Data":"4d1b6a0958405e70a9f74d458070972d7112ad956c090e4ac078281e0474474a"} Oct 30 21:43:48.920627 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:48.920592 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:43:49.077271 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:49.077202 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:38:48 +0000 UTC" deadline="2028-08-15 09:02:06.640869064 +0000 UTC" Oct 30 21:43:49.077271 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:49.077250 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15707h18m17.563632819s" Oct 30 21:43:49.178632 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:49.178607 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fe5a17_9648_48a9_8d1b_338990dac55b.slice/crio-1331dedb0706dd8e54256d0095d902066067630520984491baf848e5588d3968 WatchSource:0}: Error finding container 1331dedb0706dd8e54256d0095d902066067630520984491baf848e5588d3968: Status 404 returned error can't find the container with id 1331dedb0706dd8e54256d0095d902066067630520984491baf848e5588d3968 Oct 30 21:43:49.259366 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:49.259333 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac3165a_2ba0_4be9_8ab9_4d088d9a1432.slice/crio-bf792177a2b8333824e37fc3a6ada83da994aa5a4ed790617bb8bba444ad52c6 WatchSource:0}: Error finding container bf792177a2b8333824e37fc3a6ada83da994aa5a4ed790617bb8bba444ad52c6: Status 404 returned error can't find the container with id bf792177a2b8333824e37fc3a6ada83da994aa5a4ed790617bb8bba444ad52c6 Oct 30 21:43:49.260096 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:49.260076 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee354605_03c9_4c69_a937_5f94440dc0ae.slice/crio-37442a05753618fe8f9a34c180b2490a39fb6babbff7a4e0414d23e01cd90209 WatchSource:0}: Error finding container 37442a05753618fe8f9a34c180b2490a39fb6babbff7a4e0414d23e01cd90209: Status 404 returned error can't find the container with id 37442a05753618fe8f9a34c180b2490a39fb6babbff7a4e0414d23e01cd90209 Oct 30 21:43:49.277622 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:43:49.277598 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a34a83c_e6d4_42c4_8bd6_0c39fbcdf535.slice/crio-ab84e6c2898e89b5c29be5740b8bde81a420d3c21761cd0804d7fa890f1c4bfd WatchSource:0}: Error finding container ab84e6c2898e89b5c29be5740b8bde81a420d3c21761cd0804d7fa890f1c4bfd: Status 404 returned error can't find the container with id ab84e6c2898e89b5c29be5740b8bde81a420d3c21761cd0804d7fa890f1c4bfd Oct 30 21:43:49.647313 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:49.647279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:49.647500 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:49.647334 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs\") pod \"network-metrics-daemon-wh7rd\" (UID: \"c94b6070-d303-4c76-8895-73cd37a2f985\") " pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:43:49.647500 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:49.647450 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:43:49.647617 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:49.647510 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs podName:c94b6070-d303-4c76-8895-73cd37a2f985 nodeName:}" failed. No retries permitted until 2026-10-30 21:43:51.647490953 +0000 UTC m=+5.292069915 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs") pod "network-metrics-daemon-wh7rd" (UID: "c94b6070-d303-4c76-8895-73cd37a2f985") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:43:49.647617 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:49.647572 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:43:49.647617 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:49.647609 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret podName:f73329a3-7290-44a1-a72e-1fe72c0d0da5 nodeName:}" failed. No retries permitted until 2026-10-30 21:43:51.647596157 +0000 UTC m=+5.292175126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret") pod "global-pull-secret-syncer-8mr69" (UID: "f73329a3-7290-44a1-a72e-1fe72c0d0da5") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:43:49.748303 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:49.748266 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cgbv\" (UniqueName: \"kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv\") pod \"network-check-target-tcckj\" (UID: \"3ee9e6d0-f122-4ff3-aa72-b597339928ac\") " pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:43:49.748814 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:49.748448 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:43:49.748814 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:49.748470 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:43:49.748814 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:49.748484 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5cgbv for pod openshift-network-diagnostics/network-check-target-tcckj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:43:49.748814 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:49.748538 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv podName:3ee9e6d0-f122-4ff3-aa72-b597339928ac nodeName:}" failed. No retries permitted until 2026-10-30 21:43:51.748519766 +0000 UTC m=+5.393098741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5cgbv" (UniqueName: "kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv") pod "network-check-target-tcckj" (UID: "3ee9e6d0-f122-4ff3-aa72-b597339928ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:43:49.796903 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:49.796870 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:43:49.797054 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:49.797012 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcckj" podUID="3ee9e6d0-f122-4ff3-aa72-b597339928ac" Oct 30 21:43:49.797122 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:49.797111 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:43:49.797225 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:49.797202 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wh7rd" podUID="c94b6070-d303-4c76-8895-73cd37a2f985" Oct 30 21:43:49.797289 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:49.797271 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:49.797365 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:49.797344 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8mr69" podUID="f73329a3-7290-44a1-a72e-1fe72c0d0da5" Oct 30 21:43:49.820950 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:49.820919 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vkl9r" event={"ID":"4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535","Type":"ContainerStarted","Data":"ab84e6c2898e89b5c29be5740b8bde81a420d3c21761cd0804d7fa890f1c4bfd"} Oct 30 21:43:49.823324 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:49.823292 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-45jdt" event={"ID":"ee354605-03c9-4c69-a937-5f94440dc0ae","Type":"ContainerStarted","Data":"37442a05753618fe8f9a34c180b2490a39fb6babbff7a4e0414d23e01cd90209"} Oct 30 21:43:49.830340 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:49.830317 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" event={"ID":"08685d97-2f18-4abe-a7b8-25466aee77cb","Type":"ContainerStarted","Data":"cc8f0960d9708f19011021a19fbce978ac0df060b9f68e9b9086b32cea43803b"} Oct 30 21:43:49.832565 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:49.832476 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-89cpv" event={"ID":"74fe5a17-9648-48a9-8d1b-338990dac55b","Type":"ContainerStarted","Data":"1331dedb0706dd8e54256d0095d902066067630520984491baf848e5588d3968"} Oct 30 21:43:49.836166 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:49.836138 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6b4f" event={"ID":"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432","Type":"ContainerStarted","Data":"bf792177a2b8333824e37fc3a6ada83da994aa5a4ed790617bb8bba444ad52c6"} Oct 30 21:43:50.077462 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:50.077350 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:38:48 +0000 UTC" deadline="2028-06-24 23:27:27.991904212 +0000 UTC" Oct 30 21:43:50.077462 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:50.077417 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14473h43m37.91449267s" Oct 30 21:43:51.663893 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:51.663855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:51.664397 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:51.663922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs\") pod \"network-metrics-daemon-wh7rd\" (UID: \"c94b6070-d303-4c76-8895-73cd37a2f985\") " pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:43:51.664397 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:51.664096 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:43:51.664397 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:51.664160 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs podName:c94b6070-d303-4c76-8895-73cd37a2f985 nodeName:}" failed. No retries permitted until 2026-10-30 21:43:55.664140534 +0000 UTC m=+9.308719496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs") pod "network-metrics-daemon-wh7rd" (UID: "c94b6070-d303-4c76-8895-73cd37a2f985") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:43:51.664397 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:51.664232 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:43:51.664397 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:51.664271 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret podName:f73329a3-7290-44a1-a72e-1fe72c0d0da5 nodeName:}" failed. No retries permitted until 2026-10-30 21:43:55.664257403 +0000 UTC m=+9.308836364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret") pod "global-pull-secret-syncer-8mr69" (UID: "f73329a3-7290-44a1-a72e-1fe72c0d0da5") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:43:51.764368 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:51.764331 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cgbv\" (UniqueName: \"kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv\") pod \"network-check-target-tcckj\" (UID: \"3ee9e6d0-f122-4ff3-aa72-b597339928ac\") " pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:43:51.764547 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:51.764527 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:43:51.764547 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:51.764546 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:43:51.764690 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:51.764560 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5cgbv for pod openshift-network-diagnostics/network-check-target-tcckj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:43:51.764690 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:51.764617 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv podName:3ee9e6d0-f122-4ff3-aa72-b597339928ac nodeName:}" failed. No retries permitted until 2026-10-30 21:43:55.764599255 +0000 UTC m=+9.409178225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5cgbv" (UniqueName: "kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv") pod "network-check-target-tcckj" (UID: "3ee9e6d0-f122-4ff3-aa72-b597339928ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:43:51.796736 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:51.796707 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:43:51.796889 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:51.796834 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcckj" podUID="3ee9e6d0-f122-4ff3-aa72-b597339928ac" Oct 30 21:43:51.796955 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:51.796905 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:43:51.797016 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:51.796999 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wh7rd" podUID="c94b6070-d303-4c76-8895-73cd37a2f985" Oct 30 21:43:51.797077 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:51.797049 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:51.797187 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:51.797122 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8mr69" podUID="f73329a3-7290-44a1-a72e-1fe72c0d0da5" Oct 30 21:43:53.797025 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:53.796761 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:43:53.797458 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:53.797044 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcckj" podUID="3ee9e6d0-f122-4ff3-aa72-b597339928ac" Oct 30 21:43:53.797458 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:53.797076 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:43:53.797458 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:53.797166 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wh7rd" podUID="c94b6070-d303-4c76-8895-73cd37a2f985" Oct 30 21:43:53.797458 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:53.797183 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:53.797458 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:53.797226 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8mr69" podUID="f73329a3-7290-44a1-a72e-1fe72c0d0da5" Oct 30 21:43:53.869400 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:53.868873 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lpx2r" event={"ID":"08df8d3b-f392-438a-81d5-3ee04ee2cd8b","Type":"ContainerStarted","Data":"eed990e0607bfbba2be596cd0f02a4233fbe00041e3970365238028f3a5293d6"} Oct 30 21:43:53.878963 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:53.878926 2576 generic.go:356] "Generic (PLEG): container finished" podID="1ac3165a-2ba0-4be9-8ab9-4d088d9a1432" containerID="c7abf33c058d51570f7345a9aa45cf0c0f230fef34da0d77274bd7e09f0cf74e" exitCode=0 Oct 30 21:43:53.879089 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:53.879030 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6b4f" event={"ID":"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432","Type":"ContainerDied","Data":"c7abf33c058d51570f7345a9aa45cf0c0f230fef34da0d77274bd7e09f0cf74e"} Oct 30 21:43:53.884899 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:53.884804 2576 generic.go:356] "Generic (PLEG): container finished" podID="de7a233ef9a012d605524a0af5a13d0b" containerID="2d3b5a1cba68397b73022adb71163b8756aedbd710b0608af9fb0e64ca9547df" exitCode=0 Oct 30 21:43:53.884899 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:53.884853 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" event={"ID":"de7a233ef9a012d605524a0af5a13d0b","Type":"ContainerDied","Data":"2d3b5a1cba68397b73022adb71163b8756aedbd710b0608af9fb0e64ca9547df"} Oct 30 21:43:53.996707 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:53.995595 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lpx2r" podStartSLOduration=2.249296812 podStartE2EDuration="6.995579779s" totalImagesPullingTime="4.746282967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:53.995537438 +0000 UTC m=+7.640116420" watchObservedRunningTime="2026-10-30 21:43:53.995579779 +0000 UTC m=+7.640158759" Oct 30 21:43:54.803145 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:54.802786 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lpx2r" Oct 30 21:43:54.803145 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:54.802895 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lpx2r" Oct 30 21:43:54.891876 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:54.891834 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal" event={"ID":"8b76742bc66466f5dd9bc77bf5370e40","Type":"ContainerStarted","Data":"75d2e0ab2086dd6af6ee94f2f8926ce0296c9046b03d6a3faf2b724517a60fad"} Oct 30 21:43:54.896560 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:54.896531 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" event={"ID":"de7a233ef9a012d605524a0af5a13d0b","Type":"ContainerStarted","Data":"4a74b113946f6d46cc889e5e67621f877d96d48fc5fc024e1421d3111959c2b0"} Oct 30 21:43:55.055628 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:55.055522 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" podStartSLOduration=8.055503277 podStartE2EDuration="8.055503277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:55.055439456 +0000 UTC m=+8.700018434" watchObservedRunningTime="2026-10-30 21:43:55.055503277 +0000 UTC m=+8.700082256" Oct 30 21:43:55.183360 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:55.183295 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal" podStartSLOduration=8.183271408 podStartE2EDuration="8.183271408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:55.182215894 +0000 UTC m=+8.826794872" watchObservedRunningTime="2026-10-30 21:43:55.183271408 +0000 UTC m=+8.827850389" Oct 30 21:43:55.697566 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:55.697490 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:55.697566 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:55.697564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs\") pod \"network-metrics-daemon-wh7rd\" (UID: \"c94b6070-d303-4c76-8895-73cd37a2f985\") " pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:43:55.697799 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:55.697734 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:43:55.697799 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:55.697792 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs podName:c94b6070-d303-4c76-8895-73cd37a2f985 nodeName:}" failed. No retries permitted until 2026-10-30 21:44:03.697775183 +0000 UTC m=+17.342354142 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs") pod "network-metrics-daemon-wh7rd" (UID: "c94b6070-d303-4c76-8895-73cd37a2f985") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:43:55.697912 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:55.697856 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:43:55.697912 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:55.697892 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret podName:f73329a3-7290-44a1-a72e-1fe72c0d0da5 nodeName:}" failed. No retries permitted until 2026-10-30 21:44:03.69788116 +0000 UTC m=+17.342460131 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret") pod "global-pull-secret-syncer-8mr69" (UID: "f73329a3-7290-44a1-a72e-1fe72c0d0da5") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:43:55.796253 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:55.796222 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:43:55.796403 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:55.796345 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcckj" podUID="3ee9e6d0-f122-4ff3-aa72-b597339928ac" Oct 30 21:43:55.796479 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:55.796417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:43:55.796533 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:55.796499 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wh7rd" podUID="c94b6070-d303-4c76-8895-73cd37a2f985" Oct 30 21:43:55.796589 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:55.796550 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:55.796665 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:55.796638 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8mr69" podUID="f73329a3-7290-44a1-a72e-1fe72c0d0da5" Oct 30 21:43:55.797980 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:55.797949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cgbv\" (UniqueName: \"kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv\") pod \"network-check-target-tcckj\" (UID: \"3ee9e6d0-f122-4ff3-aa72-b597339928ac\") " pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:43:55.798150 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:55.798135 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:43:55.798217 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:55.798162 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:43:55.798217 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:55.798175 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5cgbv for pod openshift-network-diagnostics/network-check-target-tcckj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:43:55.798374 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:55.798224 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv podName:3ee9e6d0-f122-4ff3-aa72-b597339928ac nodeName:}" failed. No retries permitted until 2026-10-30 21:44:03.798207621 +0000 UTC m=+17.442786576 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5cgbv" (UniqueName: "kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv") pod "network-check-target-tcckj" (UID: "3ee9e6d0-f122-4ff3-aa72-b597339928ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:43:55.904218 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:55.904179 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-45jdt" event={"ID":"ee354605-03c9-4c69-a937-5f94440dc0ae","Type":"ContainerStarted","Data":"692b8875021ece588105d12e6be56d013842c8a439680888722594871d7c380a"} Oct 30 21:43:55.908243 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:55.908219 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" event={"ID":"e1668e0e-55e0-493c-a735-1ff0da66eda3","Type":"ContainerStarted","Data":"796de025ea77cbae91115a3ad898f1cb878d41f8aa17c94a2df4a00ed4a322e6"} Oct 30 21:43:55.908370 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:55.908310 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:43:57.414525 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:57.414503 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:43:57.790244 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:57.790128 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:43:57.414521528Z","UUID":"bf603c6f-101c-4adf-93cb-802f3ef7a45a","Handler":null,"Name":"","Endpoint":""} Oct 30 21:43:57.792491 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:57.792471 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:43:57.792563 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:57.792504 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:43:57.796289 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:57.796234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:43:57.796289 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:57.796247 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:43:57.796462 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:57.796338 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wh7rd" podUID="c94b6070-d303-4c76-8895-73cd37a2f985" Oct 30 21:43:57.796462 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:57.796380 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:57.796462 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:57.796421 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8mr69" podUID="f73329a3-7290-44a1-a72e-1fe72c0d0da5" Oct 30 21:43:57.796462 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:57.796459 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcckj" podUID="3ee9e6d0-f122-4ff3-aa72-b597339928ac" Oct 30 21:43:57.916514 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:57.916479 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" event={"ID":"e1668e0e-55e0-493c-a735-1ff0da66eda3","Type":"ContainerStarted","Data":"c2a7882e50e2f7966939692c7d63afe0536ee5b7cd7514c2693d8bfa2335fe6d"} Oct 30 21:43:57.918746 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:57.918698 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b5fb6" event={"ID":"2a899495-aa25-4538-8eef-5640503b466a","Type":"ContainerStarted","Data":"7a6cbd7b47f367a802ed1fb7e7e75d3592534fc18e5ed64831fb7a94bf423222"} Oct 30 21:43:57.921432 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:57.921409 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vx4gh" event={"ID":"827f9fb3-0eb3-4f8f-96f2-72df80c024d8","Type":"ContainerStarted","Data":"42c01729dead1e1604161a11233cdb52608e035734d469a6535a84b0c77e7420"} Oct 30 21:43:58.038949 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:58.038722 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-b5fb6" podStartSLOduration=2.002143985 podStartE2EDuration="11.038705274s" totalImagesPullingTime="9.036561289s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:58.038493022 +0000 UTC m=+11.683072000" watchObservedRunningTime="2026-10-30 21:43:58.038705274 +0000 UTC m=+11.683284251" Oct 30 21:43:58.039478 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:58.039440 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-45jdt" podStartSLOduration=4.9390233949999995 podStartE2EDuration="11.039431507s" totalImagesPullingTime="6.100408112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:56.021183696 +0000 UTC m=+9.665762672" watchObservedRunningTime="2026-10-30 21:43:58.039431507 +0000 UTC m=+11.684010482" Oct 30 21:43:58.925678 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:58.925641 2576 generic.go:356] "Generic (PLEG): container finished" podID="1ac3165a-2ba0-4be9-8ab9-4d088d9a1432" containerID="d122bcb7e202204274baa3d7b8435ae85b2cd95774fb1f715c0ca864318a4b6a" exitCode=0 Oct 30 21:43:58.926111 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:58.925723 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6b4f" event={"ID":"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432","Type":"ContainerDied","Data":"d122bcb7e202204274baa3d7b8435ae85b2cd95774fb1f715c0ca864318a4b6a"} Oct 30 21:43:59.049730 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:59.049684 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-vx4gh" podStartSLOduration=2.9902315440000002 podStartE2EDuration="12.049670912s" totalImagesPullingTime="9.059439368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:58.154776999 +0000 UTC m=+11.799355976" watchObservedRunningTime="2026-10-30 21:43:59.049670912 +0000 UTC m=+12.694249887" Oct 30 21:43:59.796994 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:59.796958 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:43:59.796994 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:59.796990 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:43:59.797203 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:59.797086 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wh7rd" podUID="c94b6070-d303-4c76-8895-73cd37a2f985" Oct 30 21:43:59.797203 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:59.797137 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:43:59.797298 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:59.797243 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8mr69" podUID="f73329a3-7290-44a1-a72e-1fe72c0d0da5" Oct 30 21:43:59.797390 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:43:59.797365 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcckj" podUID="3ee9e6d0-f122-4ff3-aa72-b597339928ac" Oct 30 21:43:59.925125 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:59.925062 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lpx2r" Oct 30 21:43:59.925258 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:59.925177 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:43:59.926020 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:59.925999 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lpx2r" Oct 30 21:43:59.930350 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:59.930328 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vkl9r" event={"ID":"4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535","Type":"ContainerStarted","Data":"d135483241fcf257220e399b5846ecbcff87ee4e36473b413b84972fae4cf4bf"} Oct 30 21:43:59.933339 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:43:59.933315 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" event={"ID":"e1668e0e-55e0-493c-a735-1ff0da66eda3","Type":"ContainerStarted","Data":"501c6819d2cd4ec52aebcb1561be46a666ceff15899d2034ae16c9d275c77065"} Oct 30 21:44:00.162454 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:00.162411 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vkl9r" podStartSLOduration=4.450365234 podStartE2EDuration="13.162399206s" totalImagesPullingTime="8.712033972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:00.162055333 +0000 UTC m=+13.806634309" watchObservedRunningTime="2026-10-30 21:44:00.162399206 +0000 UTC m=+13.806978200" Oct 30 21:44:00.281206 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:00.281125 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" podStartSLOduration=3.289526224 podStartE2EDuration="13.281110171s" totalImagesPullingTime="9.991583947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:43:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:00.28054278 +0000 UTC m=+13.925121758" watchObservedRunningTime="2026-10-30 21:44:00.281110171 +0000 UTC m=+13.925689147" Oct 30 21:44:00.936803 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:00.936722 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-89cpv" event={"ID":"74fe5a17-9648-48a9-8d1b-338990dac55b","Type":"ContainerStarted","Data":"9a63b2bc4f807b1bb920aa8bef72e86d624062516a0f407c3aa34edb9f9ceb2f"} Oct 30 21:44:00.939177 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:00.939150 2576 generic.go:356] "Generic (PLEG): container finished" podID="1ac3165a-2ba0-4be9-8ab9-4d088d9a1432" containerID="77aaa48ef6d2906233a336132bd2c050cb5abbe4d48158ce23353b5f594b4255" exitCode=0 Oct 30 21:44:00.939314 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:00.939242 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6b4f" event={"ID":"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432","Type":"ContainerDied","Data":"77aaa48ef6d2906233a336132bd2c050cb5abbe4d48158ce23353b5f594b4255"} Oct 30 21:44:01.054715 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:01.054673 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-89cpv" podStartSLOduration=2.918160104 podStartE2EDuration="14.054660844s" totalImagesPullingTime="11.13650074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:01.05460288 +0000 UTC m=+14.699181856" watchObservedRunningTime="2026-10-30 21:44:01.054660844 +0000 UTC m=+14.699239818" Oct 30 21:44:01.796267 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:01.796238 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:44:01.796448 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:01.796365 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:01.796448 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:01.796383 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcckj" podUID="3ee9e6d0-f122-4ff3-aa72-b597339928ac" Oct 30 21:44:01.796448 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:01.796411 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:01.796652 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:01.796532 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wh7rd" podUID="c94b6070-d303-4c76-8895-73cd37a2f985" Oct 30 21:44:01.796652 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:01.796608 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8mr69" podUID="f73329a3-7290-44a1-a72e-1fe72c0d0da5" Oct 30 21:44:02.947036 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:02.946838 2576 generic.go:356] "Generic (PLEG): container finished" podID="1ac3165a-2ba0-4be9-8ab9-4d088d9a1432" containerID="8e1afa093cafdc000f2e4e1356fde48b992c93a6f30ad67564ad4d78aea903fd" exitCode=0 Oct 30 21:44:02.947036 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:02.946919 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6b4f" event={"ID":"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432","Type":"ContainerDied","Data":"8e1afa093cafdc000f2e4e1356fde48b992c93a6f30ad67564ad4d78aea903fd"} Oct 30 21:44:03.759886 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:03.759856 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs\") pod \"network-metrics-daemon-wh7rd\" (UID: \"c94b6070-d303-4c76-8895-73cd37a2f985\") " pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:03.760073 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:03.759909 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:03.760073 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:03.760008 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:44:03.760073 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:03.760062 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs podName:c94b6070-d303-4c76-8895-73cd37a2f985 nodeName:}" failed. No retries permitted until 2026-10-30 21:44:19.760048659 +0000 UTC m=+33.404627612 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs") pod "network-metrics-daemon-wh7rd" (UID: "c94b6070-d303-4c76-8895-73cd37a2f985") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:44:03.760203 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:03.760008 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:44:03.760203 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:03.760126 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret podName:f73329a3-7290-44a1-a72e-1fe72c0d0da5 nodeName:}" failed. No retries permitted until 2026-10-30 21:44:19.760114922 +0000 UTC m=+33.404693889 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret") pod "global-pull-secret-syncer-8mr69" (UID: "f73329a3-7290-44a1-a72e-1fe72c0d0da5") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:44:03.796193 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:03.796165 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:03.796336 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:03.796174 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:03.796336 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:03.796304 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wh7rd" podUID="c94b6070-d303-4c76-8895-73cd37a2f985" Oct 30 21:44:03.796435 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:03.796178 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:44:03.796435 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:03.796406 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8mr69" podUID="f73329a3-7290-44a1-a72e-1fe72c0d0da5" Oct 30 21:44:03.796513 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:03.796426 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcckj" podUID="3ee9e6d0-f122-4ff3-aa72-b597339928ac" Oct 30 21:44:03.861183 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:03.861158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cgbv\" (UniqueName: \"kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv\") pod \"network-check-target-tcckj\" (UID: \"3ee9e6d0-f122-4ff3-aa72-b597339928ac\") " pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:44:03.861327 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:03.861289 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:44:03.861327 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:03.861304 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:44:03.861327 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:03.861313 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5cgbv for pod openshift-network-diagnostics/network-check-target-tcckj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:44:03.861482 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:03.861361 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv podName:3ee9e6d0-f122-4ff3-aa72-b597339928ac nodeName:}" failed. No retries permitted until 2026-10-30 21:44:19.861344626 +0000 UTC m=+33.505923592 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5cgbv" (UniqueName: "kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv") pod "network-check-target-tcckj" (UID: "3ee9e6d0-f122-4ff3-aa72-b597339928ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:44:04.955198 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:04.955031 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" event={"ID":"08685d97-2f18-4abe-a7b8-25466aee77cb","Type":"ContainerStarted","Data":"d5218c6fdcf26506b6632a7b03a2e086ea1c8e85af4584c5d5aea4e9008474c0"} Oct 30 21:44:04.955652 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:04.955251 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" event={"ID":"08685d97-2f18-4abe-a7b8-25466aee77cb","Type":"ContainerStarted","Data":"527c94c4ca7149b9ad941a815826138f7764d38bd411b73627c789062671881b"} Oct 30 21:44:04.955652 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:04.955306 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" event={"ID":"08685d97-2f18-4abe-a7b8-25466aee77cb","Type":"ContainerStarted","Data":"72bdd2e8486930d3c2cab2cd58bb05f04eec1663e185ab4a6e41629cf56942fa"} Oct 30 21:44:04.955652 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:04.955357 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" event={"ID":"08685d97-2f18-4abe-a7b8-25466aee77cb","Type":"ContainerStarted","Data":"fabe776fd5cfacaf01d155dbabb3634959a254a0d0d7b83d17116eb76f1cb6d7"} Oct 30 21:44:04.955652 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:04.955400 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" event={"ID":"08685d97-2f18-4abe-a7b8-25466aee77cb","Type":"ContainerStarted","Data":"c798b0dda4c94953b76714eee5172b5223d07c10fffbc24dd0afde4743bf3817"} Oct 30 21:44:04.955652 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:04.955447 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" event={"ID":"08685d97-2f18-4abe-a7b8-25466aee77cb","Type":"ContainerStarted","Data":"41e789c6debf9b98da10371af27fdc3cfade3cf338701cc1f2f21769074f142d"} Oct 30 21:44:05.796571 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:05.796543 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:05.796571 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:05.796562 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:44:05.796771 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:05.796681 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wh7rd" podUID="c94b6070-d303-4c76-8895-73cd37a2f985" Oct 30 21:44:05.796771 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:05.796731 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:05.796832 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:05.796818 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcckj" podUID="3ee9e6d0-f122-4ff3-aa72-b597339928ac" Oct 30 21:44:05.796913 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:05.796896 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8mr69" podUID="f73329a3-7290-44a1-a72e-1fe72c0d0da5" Oct 30 21:44:05.960040 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:05.960018 2576 generic.go:356] "Generic (PLEG): container finished" podID="1ac3165a-2ba0-4be9-8ab9-4d088d9a1432" containerID="95cd1c0f14efee7c6b788e4f8a3033ea3ec5ddc09cdb7c21d4748edbcc8a2e2a" exitCode=0 Oct 30 21:44:05.960455 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:05.960070 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6b4f" event={"ID":"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432","Type":"ContainerDied","Data":"95cd1c0f14efee7c6b788e4f8a3033ea3ec5ddc09cdb7c21d4748edbcc8a2e2a"} Oct 30 21:44:06.965704 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:06.965675 2576 generic.go:356] "Generic (PLEG): container finished" podID="1ac3165a-2ba0-4be9-8ab9-4d088d9a1432" containerID="ad3c2cc0ac07ab587124ca5f618ae9c52fa9dc9ea5c3e858b62913b62fb353a3" exitCode=0 Oct 30 21:44:06.966112 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:06.965736 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6b4f" event={"ID":"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432","Type":"ContainerDied","Data":"ad3c2cc0ac07ab587124ca5f618ae9c52fa9dc9ea5c3e858b62913b62fb353a3"} Oct 30 21:44:06.969263 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:06.969239 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" event={"ID":"08685d97-2f18-4abe-a7b8-25466aee77cb","Type":"ContainerStarted","Data":"fa4e8e1464295875a2afd298cc64cfa30880096153e294a2f0b7677d5c416d2d"} Oct 30 21:44:07.796817 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:07.796788 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:44:07.796944 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:07.796816 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:07.796988 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:07.796934 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcckj" podUID="3ee9e6d0-f122-4ff3-aa72-b597339928ac" Oct 30 21:44:07.797020 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:07.796987 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:07.797103 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:07.797081 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wh7rd" podUID="c94b6070-d303-4c76-8895-73cd37a2f985" Oct 30 21:44:07.797195 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:07.797172 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8mr69" podUID="f73329a3-7290-44a1-a72e-1fe72c0d0da5" Oct 30 21:44:07.975763 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:07.975733 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6b4f" event={"ID":"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432","Type":"ContainerStarted","Data":"181687b73588c9ed99a29bb58924e83945d772af98399525bfe866fce8f1230a"} Oct 30 21:44:08.100903 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:08.100827 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n6b4f" podStartSLOduration=7.966424344 podStartE2EDuration="21.100812738s" totalImagesPullingTime="13.134388394s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:43:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:08.100324686 +0000 UTC m=+21.744903662" watchObservedRunningTime="2026-10-30 21:44:08.100812738 +0000 UTC m=+21.745391712" Oct 30 21:44:09.797036 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:09.797004 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:44:09.797743 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:09.797039 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:09.797743 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:09.797045 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:09.797743 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:09.797118 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcckj" podUID="3ee9e6d0-f122-4ff3-aa72-b597339928ac" Oct 30 21:44:09.797743 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:09.797172 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wh7rd" podUID="c94b6070-d303-4c76-8895-73cd37a2f985" Oct 30 21:44:09.797743 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:09.797249 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8mr69" podUID="f73329a3-7290-44a1-a72e-1fe72c0d0da5" Oct 30 21:44:09.984352 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:09.984160 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" event={"ID":"08685d97-2f18-4abe-a7b8-25466aee77cb","Type":"ContainerStarted","Data":"5e760d94efbb84594395fea39dee9f43aa514623a6ec849d6b8957023f7de25c"} Oct 30 21:44:09.984497 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:09.984413 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:09.984497 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:09.984470 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:10.001327 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:10.001307 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:10.001439 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:10.001394 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:10.264945 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:10.264805 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" podStartSLOduration=7.960011607 podStartE2EDuration="23.264786637s" totalImagesPullingTime="15.30477503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:10.239998318 +0000 UTC m=+23.884577313" watchObservedRunningTime="2026-10-30 21:44:10.264786637 +0000 UTC m=+23.909365615" Oct 30 21:44:10.986699 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:10.986672 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:44:11.727609 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:11.727548 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8mr69"] Oct 30 21:44:11.727774 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:11.727658 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:11.727774 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:11.727741 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8mr69" podUID="f73329a3-7290-44a1-a72e-1fe72c0d0da5" Oct 30 21:44:11.731385 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:11.731357 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tcckj"] Oct 30 21:44:11.731709 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:11.731684 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:44:11.731851 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:11.731830 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcckj" podUID="3ee9e6d0-f122-4ff3-aa72-b597339928ac" Oct 30 21:44:11.777802 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:11.777776 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wh7rd"] Oct 30 21:44:11.777890 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:11.777867 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:11.777968 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:11.777950 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wh7rd" podUID="c94b6070-d303-4c76-8895-73cd37a2f985" Oct 30 21:44:11.989127 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:11.989068 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:44:12.695120 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:12.695093 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:13.796096 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:13.796073 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:13.796427 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:13.796082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:13.796427 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:13.796176 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wh7rd" podUID="c94b6070-d303-4c76-8895-73cd37a2f985" Oct 30 21:44:13.796427 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:13.796204 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:44:13.796427 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:13.796284 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8mr69" podUID="f73329a3-7290-44a1-a72e-1fe72c0d0da5" Oct 30 21:44:13.796427 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:13.796332 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcckj" podUID="3ee9e6d0-f122-4ff3-aa72-b597339928ac" Oct 30 21:44:15.796863 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:15.796830 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:44:15.797239 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:15.796908 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:15.797239 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:15.796940 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:15.797239 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:15.797051 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcckj" podUID="3ee9e6d0-f122-4ff3-aa72-b597339928ac" Oct 30 21:44:15.797239 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:15.797170 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8mr69" podUID="f73329a3-7290-44a1-a72e-1fe72c0d0da5" Oct 30 21:44:15.797409 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:15.797244 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wh7rd" podUID="c94b6070-d303-4c76-8895-73cd37a2f985" Oct 30 21:44:17.606515 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.606452 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeReady" Oct 30 21:44:17.606871 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.606569 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:44:17.760965 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.760935 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sqfth"] Oct 30 21:44:17.763868 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.763851 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tg5p8"] Oct 30 21:44:17.764027 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.764005 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:17.766893 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.766876 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tg5p8" Oct 30 21:44:17.796675 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.796649 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:17.796798 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.796656 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:44:17.796863 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.796803 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:17.799533 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.799512 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sqfth"] Oct 30 21:44:17.817019 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.817000 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:44:17.817185 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.817169 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:44:17.817317 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.817298 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qjn2k\"" Oct 30 21:44:17.819582 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.819563 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:44:17.819687 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.819662 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:44:17.819748 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.819733 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vzpnp\"" Oct 30 21:44:17.819790 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.819768 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:44:17.832730 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.832712 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tg5p8"] Oct 30 21:44:17.864249 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.864208 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/000931e4-5bd3-4387-adf7-ea47151527ad-tmp-dir\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:17.864249 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.864241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12b346ba-5a09-4ec5-8c16-885fbe1502bc-cert\") pod \"ingress-canary-tg5p8\" (UID: \"12b346ba-5a09-4ec5-8c16-885fbe1502bc\") " pod="openshift-ingress-canary/ingress-canary-tg5p8" Oct 30 21:44:17.864364 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.864259 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:17.864364 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.864343 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq6b2\" (UniqueName: \"kubernetes.io/projected/12b346ba-5a09-4ec5-8c16-885fbe1502bc-kube-api-access-pq6b2\") pod \"ingress-canary-tg5p8\" (UID: \"12b346ba-5a09-4ec5-8c16-885fbe1502bc\") " pod="openshift-ingress-canary/ingress-canary-tg5p8" Oct 30 21:44:17.864454 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.864438 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/000931e4-5bd3-4387-adf7-ea47151527ad-config-volume\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:17.864488 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.864461 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gv95\" (UniqueName: \"kubernetes.io/projected/000931e4-5bd3-4387-adf7-ea47151527ad-kube-api-access-9gv95\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:17.917900 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.917869 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:44:17.917987 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.917917 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:44:17.917987 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.917975 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:44:17.918207 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.918192 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6ptf6\"" Oct 30 21:44:17.918268 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.918212 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:44:17.918268 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.918251 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ph4bx\"" Oct 30 21:44:17.918345 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.918273 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j2rr9\"" Oct 30 21:44:17.964904 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.964884 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12b346ba-5a09-4ec5-8c16-885fbe1502bc-cert\") pod \"ingress-canary-tg5p8\" (UID: \"12b346ba-5a09-4ec5-8c16-885fbe1502bc\") " pod="openshift-ingress-canary/ingress-canary-tg5p8" Oct 30 21:44:17.964985 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.964909 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:17.964985 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.964941 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq6b2\" (UniqueName: \"kubernetes.io/projected/12b346ba-5a09-4ec5-8c16-885fbe1502bc-kube-api-access-pq6b2\") pod \"ingress-canary-tg5p8\" (UID: \"12b346ba-5a09-4ec5-8c16-885fbe1502bc\") " pod="openshift-ingress-canary/ingress-canary-tg5p8" Oct 30 21:44:17.965087 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:17.965065 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:44:17.965136 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:17.965123 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls podName:000931e4-5bd3-4387-adf7-ea47151527ad nodeName:}" failed. No retries permitted until 2026-10-30 21:44:18.465103309 +0000 UTC m=+32.109682275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls") pod "dns-default-sqfth" (UID: "000931e4-5bd3-4387-adf7-ea47151527ad") : secret "dns-default-metrics-tls" not found Oct 30 21:44:17.965190 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.965148 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/000931e4-5bd3-4387-adf7-ea47151527ad-config-volume\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:17.965190 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.965181 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gv95\" (UniqueName: \"kubernetes.io/projected/000931e4-5bd3-4387-adf7-ea47151527ad-kube-api-access-9gv95\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:17.965282 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.965207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/000931e4-5bd3-4387-adf7-ea47151527ad-tmp-dir\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:17.965552 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.965526 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/000931e4-5bd3-4387-adf7-ea47151527ad-tmp-dir\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:17.965796 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.965778 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/000931e4-5bd3-4387-adf7-ea47151527ad-config-volume\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:17.969296 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:17.969178 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12b346ba-5a09-4ec5-8c16-885fbe1502bc-cert\") pod \"ingress-canary-tg5p8\" (UID: \"12b346ba-5a09-4ec5-8c16-885fbe1502bc\") " pod="openshift-ingress-canary/ingress-canary-tg5p8" Oct 30 21:44:18.026780 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.026752 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq6b2\" (UniqueName: \"kubernetes.io/projected/12b346ba-5a09-4ec5-8c16-885fbe1502bc-kube-api-access-pq6b2\") pod \"ingress-canary-tg5p8\" (UID: \"12b346ba-5a09-4ec5-8c16-885fbe1502bc\") " pod="openshift-ingress-canary/ingress-canary-tg5p8" Oct 30 21:44:18.026780 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.026761 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gv95\" (UniqueName: \"kubernetes.io/projected/000931e4-5bd3-4387-adf7-ea47151527ad-kube-api-access-9gv95\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:18.082406 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.082390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tg5p8" Oct 30 21:44:18.345664 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.345637 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f8pkp"] Oct 30 21:44:18.354273 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.354257 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:18.407592 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.407543 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f8pkp"] Oct 30 21:44:18.459541 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.459516 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tg5p8"] Oct 30 21:44:18.462954 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:44:18.462935 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b346ba_5a09_4ec5_8c16_885fbe1502bc.slice/crio-7be60c6591fc4f959088b7fe0725ed1b1749b5bb4d768ef330aa5e72d0671a76 WatchSource:0}: Error finding container 7be60c6591fc4f959088b7fe0725ed1b1749b5bb4d768ef330aa5e72d0671a76: Status 404 returned error can't find the container with id 7be60c6591fc4f959088b7fe0725ed1b1749b5bb4d768ef330aa5e72d0671a76 Oct 30 21:44:18.467781 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.467751 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92da1846-1753-43a7-84a7-3f317e84675d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:18.467864 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.467790 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7k9p\" (UniqueName: \"kubernetes.io/projected/92da1846-1753-43a7-84a7-3f317e84675d-kube-api-access-m7k9p\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:18.467926 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.467883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:18.467979 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.467943 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:18.468021 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:18.467987 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:44:18.468082 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.468062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92da1846-1753-43a7-84a7-3f317e84675d-crio-socket\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:18.468123 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:18.468081 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls podName:000931e4-5bd3-4387-adf7-ea47151527ad nodeName:}" failed. No retries permitted until 2026-10-30 21:44:19.468066004 +0000 UTC m=+33.112644975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls") pod "dns-default-sqfth" (UID: "000931e4-5bd3-4387-adf7-ea47151527ad") : secret "dns-default-metrics-tls" not found Oct 30 21:44:18.468123 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.468105 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92da1846-1753-43a7-84a7-3f317e84675d-data-volume\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:18.503531 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.503507 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:44:18.503531 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.503514 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:44:18.503746 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.503729 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:44:18.503813 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.503749 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:44:18.503813 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.503772 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ltjsw\"" Oct 30 21:44:18.569329 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.569305 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92da1846-1753-43a7-84a7-3f317e84675d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:18.569420 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.569335 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7k9p\" (UniqueName: \"kubernetes.io/projected/92da1846-1753-43a7-84a7-3f317e84675d-kube-api-access-m7k9p\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:18.569420 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.569370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:18.569525 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:18.569448 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:44:18.569525 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.569466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92da1846-1753-43a7-84a7-3f317e84675d-crio-socket\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:18.569525 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:18.569481 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls podName:92da1846-1753-43a7-84a7-3f317e84675d nodeName:}" failed. No retries permitted until 2026-10-30 21:44:19.069470999 +0000 UTC m=+32.714049952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8pkp" (UID: "92da1846-1753-43a7-84a7-3f317e84675d") : secret "insights-runtime-extractor-tls" not found Oct 30 21:44:18.569525 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.569499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92da1846-1753-43a7-84a7-3f317e84675d-data-volume\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:18.569727 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:18.569545 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:44:18.569727 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.569669 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92da1846-1753-43a7-84a7-3f317e84675d-crio-socket\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:18.569727 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:18.569717 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92da1846-1753-43a7-84a7-3f317e84675d-kube-rbac-proxy-cm podName:92da1846-1753-43a7-84a7-3f317e84675d nodeName:}" failed. No retries permitted until 2026-10-30 21:44:19.069693347 +0000 UTC m=+32.714272320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92da1846-1753-43a7-84a7-3f317e84675d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8pkp" (UID: "92da1846-1753-43a7-84a7-3f317e84675d") : configmap "kube-rbac-proxy" not found Oct 30 21:44:18.569841 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.569796 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92da1846-1753-43a7-84a7-3f317e84675d-data-volume\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:18.584205 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.584182 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-bcs5k-f5hhp"] Oct 30 21:44:18.590314 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.590272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:18.631440 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.631418 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7k9p\" (UniqueName: \"kubernetes.io/projected/92da1846-1753-43a7-84a7-3f317e84675d-kube-api-access-m7k9p\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:18.643365 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.643342 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:44:18.643584 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.643571 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-f7x6x\"" Oct 30 21:44:18.644433 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.644415 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bcs5k-f5hhp"] Oct 30 21:44:18.670570 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.670525 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-trusted-ca-bundle\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:18.670570 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.670560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d45916e8-c498-4bf5-b5f3-df697b608ed9-archives-path\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:18.670699 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.670592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrgmx\" (UniqueName: \"kubernetes.io/projected/d45916e8-c498-4bf5-b5f3-df697b608ed9-kube-api-access-lrgmx\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:18.670736 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.670696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-service-ca-bundle\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:18.695117 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.695099 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:44:18.771283 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.771258 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrgmx\" (UniqueName: \"kubernetes.io/projected/d45916e8-c498-4bf5-b5f3-df697b608ed9-kube-api-access-lrgmx\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:18.771367 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.771297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-service-ca-bundle\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:18.771404 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.771361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-trusted-ca-bundle\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:18.771447 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.771414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d45916e8-c498-4bf5-b5f3-df697b608ed9-archives-path\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:18.771818 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.771802 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d45916e8-c498-4bf5-b5f3-df697b608ed9-archives-path\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:18.771925 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.771908 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-service-ca-bundle\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:18.772353 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.772337 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-trusted-ca-bundle\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:18.832705 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.832683 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrgmx\" (UniqueName: \"kubernetes.io/projected/d45916e8-c498-4bf5-b5f3-df697b608ed9-kube-api-access-lrgmx\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:18.900070 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:18.900049 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:19.004987 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:19.004963 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tg5p8" event={"ID":"12b346ba-5a09-4ec5-8c16-885fbe1502bc","Type":"ContainerStarted","Data":"7be60c6591fc4f959088b7fe0725ed1b1749b5bb4d768ef330aa5e72d0671a76"} Oct 30 21:44:19.073312 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:19.073283 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92da1846-1753-43a7-84a7-3f317e84675d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:19.073409 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:19.073353 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:19.073409 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:19.073396 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:44:19.073527 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:19.073454 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92da1846-1753-43a7-84a7-3f317e84675d-kube-rbac-proxy-cm podName:92da1846-1753-43a7-84a7-3f317e84675d nodeName:}" failed. No retries permitted until 2026-10-30 21:44:20.073440541 +0000 UTC m=+33.718019519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92da1846-1753-43a7-84a7-3f317e84675d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8pkp" (UID: "92da1846-1753-43a7-84a7-3f317e84675d") : configmap "kube-rbac-proxy" not found Oct 30 21:44:19.073527 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:19.073464 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:44:19.073527 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:19.073517 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls podName:92da1846-1753-43a7-84a7-3f317e84675d nodeName:}" failed. No retries permitted until 2026-10-30 21:44:20.073501539 +0000 UTC m=+33.718080508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8pkp" (UID: "92da1846-1753-43a7-84a7-3f317e84675d") : secret "insights-runtime-extractor-tls" not found Oct 30 21:44:19.302797 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:19.302748 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bcs5k-f5hhp"] Oct 30 21:44:19.305120 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:44:19.305096 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45916e8_c498_4bf5_b5f3_df697b608ed9.slice/crio-55c67840b3a6b6d25f6059d282483d6436754966dd76cadfbab8fba03c0c5f16 WatchSource:0}: Error finding container 55c67840b3a6b6d25f6059d282483d6436754966dd76cadfbab8fba03c0c5f16: Status 404 returned error can't find the container with id 55c67840b3a6b6d25f6059d282483d6436754966dd76cadfbab8fba03c0c5f16 Oct 30 21:44:19.476940 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:19.476907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:19.477105 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:19.477076 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:44:19.477236 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:19.477145 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls podName:000931e4-5bd3-4387-adf7-ea47151527ad nodeName:}" failed. No retries permitted until 2026-10-30 21:44:21.477125943 +0000 UTC m=+35.121704919 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls") pod "dns-default-sqfth" (UID: "000931e4-5bd3-4387-adf7-ea47151527ad") : secret "dns-default-metrics-tls" not found Oct 30 21:44:19.778353 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:19.778281 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:19.778353 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:19.778348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs\") pod \"network-metrics-daemon-wh7rd\" (UID: \"c94b6070-d303-4c76-8895-73cd37a2f985\") " pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:19.778786 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:19.778492 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:44:19.778786 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:19.778554 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs podName:c94b6070-d303-4c76-8895-73cd37a2f985 nodeName:}" failed. No retries permitted until 2026-10-30 21:44:51.778537125 +0000 UTC m=+65.423116097 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs") pod "network-metrics-daemon-wh7rd" (UID: "c94b6070-d303-4c76-8895-73cd37a2f985") : secret "metrics-daemon-secret" not found Oct 30 21:44:19.781027 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:19.781002 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:19.879527 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:19.879491 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cgbv\" (UniqueName: \"kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv\") pod \"network-check-target-tcckj\" (UID: \"3ee9e6d0-f122-4ff3-aa72-b597339928ac\") " pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:44:19.882071 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:19.882049 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cgbv\" (UniqueName: \"kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv\") pod \"network-check-target-tcckj\" (UID: \"3ee9e6d0-f122-4ff3-aa72-b597339928ac\") " pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:44:19.910837 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:19.910816 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:19.916005 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:19.915974 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:44:19.999827 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:19.999796 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nxf92"] Oct 30 21:44:20.033713 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.033651 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" event={"ID":"d45916e8-c498-4bf5-b5f3-df697b608ed9","Type":"ContainerStarted","Data":"55c67840b3a6b6d25f6059d282483d6436754966dd76cadfbab8fba03c0c5f16"} Oct 30 21:44:20.033811 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.033767 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxf92" Oct 30 21:44:20.060360 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.060334 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nxf92"] Oct 30 21:44:20.081671 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.081650 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:20.081782 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.081691 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvscc\" (UniqueName: \"kubernetes.io/projected/3a846703-92eb-4115-875f-c347fc4f2a1c-kube-api-access-tvscc\") pod \"migrator-66df5d496b-nxf92\" (UID: \"3a846703-92eb-4115-875f-c347fc4f2a1c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxf92" Oct 30 21:44:20.081782 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.081734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92da1846-1753-43a7-84a7-3f317e84675d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:20.081896 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:20.081831 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:44:20.081896 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:20.081847 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:44:20.081896 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:20.081894 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92da1846-1753-43a7-84a7-3f317e84675d-kube-rbac-proxy-cm podName:92da1846-1753-43a7-84a7-3f317e84675d nodeName:}" failed. No retries permitted until 2026-10-30 21:44:22.081878199 +0000 UTC m=+35.726457167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92da1846-1753-43a7-84a7-3f317e84675d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8pkp" (UID: "92da1846-1753-43a7-84a7-3f317e84675d") : configmap "kube-rbac-proxy" not found Oct 30 21:44:20.082012 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:20.081907 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls podName:92da1846-1753-43a7-84a7-3f317e84675d nodeName:}" failed. No retries permitted until 2026-10-30 21:44:22.081901774 +0000 UTC m=+35.726480727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8pkp" (UID: "92da1846-1753-43a7-84a7-3f317e84675d") : secret "insights-runtime-extractor-tls" not found Oct 30 21:44:20.088032 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.088006 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:44:20.088140 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.088076 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:44:20.088332 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.088310 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sqd9j\"" Oct 30 21:44:20.182267 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.182240 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvscc\" (UniqueName: \"kubernetes.io/projected/3a846703-92eb-4115-875f-c347fc4f2a1c-kube-api-access-tvscc\") pod \"migrator-66df5d496b-nxf92\" (UID: \"3a846703-92eb-4115-875f-c347fc4f2a1c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxf92" Oct 30 21:44:20.244378 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.244352 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvscc\" (UniqueName: \"kubernetes.io/projected/3a846703-92eb-4115-875f-c347fc4f2a1c-kube-api-access-tvscc\") pod \"migrator-66df5d496b-nxf92\" (UID: \"3a846703-92eb-4115-875f-c347fc4f2a1c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxf92" Oct 30 21:44:20.306431 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.306362 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8mr69"] Oct 30 21:44:20.309479 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:44:20.309453 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73329a3_7290_44a1_a72e_1fe72c0d0da5.slice/crio-196cde0e820e2f4d7dfd3a3316445ad7a192fd2fb3f5559fe367a5d575cd2956 WatchSource:0}: Error finding container 196cde0e820e2f4d7dfd3a3316445ad7a192fd2fb3f5559fe367a5d575cd2956: Status 404 returned error can't find the container with id 196cde0e820e2f4d7dfd3a3316445ad7a192fd2fb3f5559fe367a5d575cd2956 Oct 30 21:44:20.356811 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.356789 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tcckj"] Oct 30 21:44:20.358733 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.358713 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxf92" Oct 30 21:44:20.360680 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:44:20.360649 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee9e6d0_f122_4ff3_aa72_b597339928ac.slice/crio-b151e7777a44f9a184b6ba2b452dbd24cf00f56bb074af9d289768464a80be0e WatchSource:0}: Error finding container b151e7777a44f9a184b6ba2b452dbd24cf00f56bb074af9d289768464a80be0e: Status 404 returned error can't find the container with id b151e7777a44f9a184b6ba2b452dbd24cf00f56bb074af9d289768464a80be0e Oct 30 21:44:20.509397 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.509370 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-2p9r5"] Oct 30 21:44:20.527624 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.527587 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" Oct 30 21:44:20.570928 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.570876 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-2p9r5"] Oct 30 21:44:20.581241 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.580914 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-4bwnp\"" Oct 30 21:44:20.581241 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.581032 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 21:44:20.581241 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.581069 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 21:44:20.581241 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.581132 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 21:44:20.581241 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.581202 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 21:44:20.587691 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.587666 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8f03953c-1f9f-4b43-8f1d-0a5f3766340b-signing-key\") pod \"service-ca-5fffc97bdf-2p9r5\" (UID: \"8f03953c-1f9f-4b43-8f1d-0a5f3766340b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" Oct 30 21:44:20.587791 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.587707 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64p8r\" (UniqueName: \"kubernetes.io/projected/8f03953c-1f9f-4b43-8f1d-0a5f3766340b-kube-api-access-64p8r\") pod \"service-ca-5fffc97bdf-2p9r5\" (UID: \"8f03953c-1f9f-4b43-8f1d-0a5f3766340b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" Oct 30 21:44:20.587858 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.587805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8f03953c-1f9f-4b43-8f1d-0a5f3766340b-signing-cabundle\") pod \"service-ca-5fffc97bdf-2p9r5\" (UID: \"8f03953c-1f9f-4b43-8f1d-0a5f3766340b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" Oct 30 21:44:20.689166 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.689001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8f03953c-1f9f-4b43-8f1d-0a5f3766340b-signing-key\") pod \"service-ca-5fffc97bdf-2p9r5\" (UID: \"8f03953c-1f9f-4b43-8f1d-0a5f3766340b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" Oct 30 21:44:20.689285 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.689186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64p8r\" (UniqueName: \"kubernetes.io/projected/8f03953c-1f9f-4b43-8f1d-0a5f3766340b-kube-api-access-64p8r\") pod \"service-ca-5fffc97bdf-2p9r5\" (UID: \"8f03953c-1f9f-4b43-8f1d-0a5f3766340b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" Oct 30 21:44:20.689380 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.689356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8f03953c-1f9f-4b43-8f1d-0a5f3766340b-signing-cabundle\") pod \"service-ca-5fffc97bdf-2p9r5\" (UID: \"8f03953c-1f9f-4b43-8f1d-0a5f3766340b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" Oct 30 21:44:20.690048 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.690024 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8f03953c-1f9f-4b43-8f1d-0a5f3766340b-signing-cabundle\") pod \"service-ca-5fffc97bdf-2p9r5\" (UID: \"8f03953c-1f9f-4b43-8f1d-0a5f3766340b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" Oct 30 21:44:20.692460 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.692441 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8f03953c-1f9f-4b43-8f1d-0a5f3766340b-signing-key\") pod \"service-ca-5fffc97bdf-2p9r5\" (UID: \"8f03953c-1f9f-4b43-8f1d-0a5f3766340b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" Oct 30 21:44:20.751835 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.751803 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64p8r\" (UniqueName: \"kubernetes.io/projected/8f03953c-1f9f-4b43-8f1d-0a5f3766340b-kube-api-access-64p8r\") pod \"service-ca-5fffc97bdf-2p9r5\" (UID: \"8f03953c-1f9f-4b43-8f1d-0a5f3766340b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" Oct 30 21:44:20.755501 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.755476 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nxf92"] Oct 30 21:44:20.759382 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:44:20.759341 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a846703_92eb_4115_875f_c347fc4f2a1c.slice/crio-eb4d4ca5723228ea9975e335fa64c10fe04d537c57c0dbe7f87209cbc3b2c958 WatchSource:0}: Error finding container eb4d4ca5723228ea9975e335fa64c10fe04d537c57c0dbe7f87209cbc3b2c958: Status 404 returned error can't find the container with id eb4d4ca5723228ea9975e335fa64c10fe04d537c57c0dbe7f87209cbc3b2c958 Oct 30 21:44:20.851979 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:20.851919 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" Oct 30 21:44:21.014035 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:21.013999 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tcckj" event={"ID":"3ee9e6d0-f122-4ff3-aa72-b597339928ac","Type":"ContainerStarted","Data":"b151e7777a44f9a184b6ba2b452dbd24cf00f56bb074af9d289768464a80be0e"} Oct 30 21:44:21.015507 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:21.015465 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxf92" event={"ID":"3a846703-92eb-4115-875f-c347fc4f2a1c","Type":"ContainerStarted","Data":"eb4d4ca5723228ea9975e335fa64c10fe04d537c57c0dbe7f87209cbc3b2c958"} Oct 30 21:44:21.016877 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:21.016856 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8mr69" event={"ID":"f73329a3-7290-44a1-a72e-1fe72c0d0da5","Type":"ContainerStarted","Data":"196cde0e820e2f4d7dfd3a3316445ad7a192fd2fb3f5559fe367a5d575cd2956"} Oct 30 21:44:21.271031 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:21.270929 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-2p9r5"] Oct 30 21:44:21.280889 ip-10-0-131-193 kubenswrapper[2576]: W1030 21:44:21.280856 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f03953c_1f9f_4b43_8f1d_0a5f3766340b.slice/crio-052ec105874a388a6268608054e30666c883b13aef72d2e9ab2de4d7b5da36fd WatchSource:0}: Error finding container 052ec105874a388a6268608054e30666c883b13aef72d2e9ab2de4d7b5da36fd: Status 404 returned error can't find the container with id 052ec105874a388a6268608054e30666c883b13aef72d2e9ab2de4d7b5da36fd Oct 30 21:44:21.495743 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:21.495709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:21.495924 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:21.495886 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:44:21.495989 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:21.495944 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls podName:000931e4-5bd3-4387-adf7-ea47151527ad nodeName:}" failed. No retries permitted until 2026-10-30 21:44:25.495927464 +0000 UTC m=+39.140506436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls") pod "dns-default-sqfth" (UID: "000931e4-5bd3-4387-adf7-ea47151527ad") : secret "dns-default-metrics-tls" not found Oct 30 21:44:22.023241 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:22.023198 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" event={"ID":"8f03953c-1f9f-4b43-8f1d-0a5f3766340b","Type":"ContainerStarted","Data":"052ec105874a388a6268608054e30666c883b13aef72d2e9ab2de4d7b5da36fd"} Oct 30 21:44:22.100359 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:22.100308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92da1846-1753-43a7-84a7-3f317e84675d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:22.100587 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:22.100399 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:22.100587 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:22.100458 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:44:22.100587 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:22.100532 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92da1846-1753-43a7-84a7-3f317e84675d-kube-rbac-proxy-cm podName:92da1846-1753-43a7-84a7-3f317e84675d nodeName:}" failed. No retries permitted until 2026-10-30 21:44:26.100512446 +0000 UTC m=+39.745091402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92da1846-1753-43a7-84a7-3f317e84675d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8pkp" (UID: "92da1846-1753-43a7-84a7-3f317e84675d") : configmap "kube-rbac-proxy" not found Oct 30 21:44:22.100587 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:22.100547 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:44:22.100863 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:22.100600 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls podName:92da1846-1753-43a7-84a7-3f317e84675d nodeName:}" failed. No retries permitted until 2026-10-30 21:44:26.100584432 +0000 UTC m=+39.745163386 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8pkp" (UID: "92da1846-1753-43a7-84a7-3f317e84675d") : secret "insights-runtime-extractor-tls" not found Oct 30 21:44:23.028730 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:23.027995 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" event={"ID":"d45916e8-c498-4bf5-b5f3-df697b608ed9","Type":"ContainerStarted","Data":"3af588a860ebfaa70bb64808d17d784b73849d688080c5d46ecb233ac04fdeb2"} Oct 30 21:44:23.031288 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:23.031253 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tg5p8" event={"ID":"12b346ba-5a09-4ec5-8c16-885fbe1502bc","Type":"ContainerStarted","Data":"22de34c4d845b7d554dc8307a22c2478ade6a90b92f024acdc8118cd07cc7dd7"} Oct 30 21:44:23.148212 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:23.148161 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" podStartSLOduration=1.771373227 podStartE2EDuration="5.148148881s" totalImagesPullingTime="3.376775654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:23.147876236 +0000 UTC m=+36.792455212" watchObservedRunningTime="2026-10-30 21:44:23.148148881 +0000 UTC m=+36.792727850" Oct 30 21:44:24.036079 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:24.035973 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxf92" event={"ID":"3a846703-92eb-4115-875f-c347fc4f2a1c","Type":"ContainerStarted","Data":"6aec1021a1cbe8d8a6dd99ab507d142131a0c0c7a880ed4b39df9555b9fa801f"} Oct 30 21:44:24.152833 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:24.152776 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxf92" podStartSLOduration=2.851681479 podStartE2EDuration="5.152760598s" totalImagesPullingTime="2.301079119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:24.15250152 +0000 UTC m=+37.797080496" watchObservedRunningTime="2026-10-30 21:44:24.152760598 +0000 UTC m=+37.797339574" Oct 30 21:44:24.153430 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:24.153391 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tg5p8" podStartSLOduration=3.498655787 podStartE2EDuration="7.153382725s" totalImagesPullingTime="3.654726938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:23.266674104 +0000 UTC m=+36.911253079" watchObservedRunningTime="2026-10-30 21:44:24.153382725 +0000 UTC m=+37.797961700" Oct 30 21:44:25.534117 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:25.533851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:25.534117 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:25.534107 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:44:25.534740 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:25.534182 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls podName:000931e4-5bd3-4387-adf7-ea47151527ad nodeName:}" failed. No retries permitted until 2026-10-30 21:44:33.534163173 +0000 UTC m=+47.178742149 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls") pod "dns-default-sqfth" (UID: "000931e4-5bd3-4387-adf7-ea47151527ad") : secret "dns-default-metrics-tls" not found Oct 30 21:44:26.043815 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:26.043773 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tcckj" event={"ID":"3ee9e6d0-f122-4ff3-aa72-b597339928ac","Type":"ContainerStarted","Data":"37b7566e6629543cc55d3b516b81c7f51bc56ea65771a41e8e56f6e6b77a1a34"} Oct 30 21:44:26.043994 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:26.043857 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:44:26.045661 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:26.045633 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" event={"ID":"8f03953c-1f9f-4b43-8f1d-0a5f3766340b","Type":"ContainerStarted","Data":"1aec177049327b81aa30b58887ea2052e2a982e2a74b74820ded9c0fb10e9bda"} Oct 30 21:44:26.140145 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:26.140119 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92da1846-1753-43a7-84a7-3f317e84675d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:26.140277 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:26.140185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:26.140419 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:26.140397 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:44:26.140497 ip-10-0-131-193 kubenswrapper[2576]: E1030 21:44:26.140485 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls podName:92da1846-1753-43a7-84a7-3f317e84675d nodeName:}" failed. No retries permitted until 2026-10-30 21:44:34.140464088 +0000 UTC m=+47.785043059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8pkp" (UID: "92da1846-1753-43a7-84a7-3f317e84675d") : secret "insights-runtime-extractor-tls" not found Oct 30 21:44:26.140837 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:26.140815 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92da1846-1753-43a7-84a7-3f317e84675d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:26.160078 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:26.160033 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tcckj" podStartSLOduration=34.354872897 podStartE2EDuration="39.16002134s" totalImagesPullingTime="4.805148443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:26.159576677 +0000 UTC m=+39.804155652" watchObservedRunningTime="2026-10-30 21:44:26.16002134 +0000 UTC m=+39.804600594" Oct 30 21:44:26.274895 ip-10-0-131-193 kubenswrapper[2576]: I1030 21:44:26.274848 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" podStartSLOduration=2.379238186 podStartE2EDuration="6.274832242s" totalImagesPullingTime="3.895594056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:26.274405428 +0000 UTC m=+39.918984402" watchObservedRunningTime="2026-10-30 21:44:26.274832242 +0000 UTC m=+39.919411218" Oct 30 21:44:26.837874 ip-10-0-131-193 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:44:26.855288 ip-10-0-131-193 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:44:26.855458 ip-10-0-131-193 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:44:26.855643 ip-10-0-131-193 systemd[1]: kubelet.service: Consumed 2.316s CPU time. Oct 30 21:44:26.868256 ip-10-0-131-193 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:44:26.962997 ip-10-0-131-193 kubenswrapper[4335]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:44:26.962997 ip-10-0-131-193 kubenswrapper[4335]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:44:26.962997 ip-10-0-131-193 kubenswrapper[4335]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:44:26.962997 ip-10-0-131-193 kubenswrapper[4335]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:44:26.965115 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965028 4335 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:44:26.965115 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965114 4335 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:44:26.965115 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965119 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965123 4335 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965126 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965128 4335 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965131 4335 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965134 4335 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965136 4335 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965139 4335 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965142 4335 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965144 4335 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965147 4335 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965149 4335 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965152 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965155 4335 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965157 4335 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965160 4335 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965162 4335 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965165 4335 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965168 4335 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:44:26.965214 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965176 4335 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965179 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965182 4335 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965185 4335 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965187 4335 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965190 4335 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965193 4335 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965195 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965198 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965201 4335 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965204 4335 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965206 4335 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965209 4335 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965212 4335 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965216 4335 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965220 4335 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965225 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965230 4335 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965238 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:44:26.965788 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965242 4335 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965247 4335 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965251 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965255 4335 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965258 4335 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965263 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965266 4335 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965270 4335 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965274 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965278 4335 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965282 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965286 4335 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965290 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965294 4335 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965298 4335 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965303 4335 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965308 4335 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965312 4335 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965315 4335 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965319 4335 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:44:26.966268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965323 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965329 4335 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965333 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965337 4335 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965342 4335 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965345 4335 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965349 4335 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965354 4335 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965358 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965362 4335 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965366 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965370 4335 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965380 4335 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965385 4335 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965389 4335 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965393 4335 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965397 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965401 4335 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965405 4335 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965409 4335 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:44:26.966781 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965413 4335 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965417 4335 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965421 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965426 4335 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965430 4335 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965433 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965436 4335 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965439 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965442 4335 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965444 4335 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965447 4335 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965450 4335 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965452 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965455 4335 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965460 4335 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965464 4335 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965467 4335 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965470 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965473 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:44:26.967268 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965479 4335 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965483 4335 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965487 4335 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965490 4335 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965493 4335 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965496 4335 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.965500 4335 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966277 4335 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966284 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966287 4335 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966290 4335 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966293 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966295 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966298 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966302 4335 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966304 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966308 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966311 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966313 4335 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966316 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:44:26.967941 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966320 4335 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966322 4335 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966325 4335 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966327 4335 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966330 4335 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966333 4335 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966335 4335 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966338 4335 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966340 4335 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966343 4335 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966346 4335 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966348 4335 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966351 4335 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966354 4335 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966357 4335 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966360 4335 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966363 4335 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966365 4335 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966368 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966371 4335 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:44:26.968457 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966374 4335 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966377 4335 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966379 4335 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966382 4335 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966384 4335 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966387 4335 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966389 4335 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966392 4335 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966395 4335 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966397 4335 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966400 4335 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966402 4335 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966406 4335 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966408 4335 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966411 4335 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966413 4335 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966416 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966420 4335 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966425 4335 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:44:26.968985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966429 4335 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966433 4335 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966436 4335 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966439 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966442 4335 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966444 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966447 4335 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966450 4335 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966453 4335 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966455 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966458 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966460 4335 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966463 4335 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966466 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966469 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966472 4335 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966474 4335 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966477 4335 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966480 4335 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966482 4335 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:44:26.969476 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966485 4335 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966487 4335 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966490 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966493 4335 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966495 4335 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966498 4335 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966500 4335 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966503 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966505 4335 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966507 4335 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966510 4335 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966512 4335 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966515 4335 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966518 4335 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966521 4335 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966523 4335 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966526 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966528 4335 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966531 4335 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966534 4335 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:44:26.969989 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966537 4335 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966539 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966542 4335 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966544 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966547 4335 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966550 4335 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966552 4335 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966556 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966559 4335 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966561 4335 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966564 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966566 4335 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966569 4335 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.966571 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967075 4335 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967081 4335 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967084 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967087 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967090 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:44:26.970516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967093 4335 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967096 4335 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967099 4335 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967101 4335 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967104 4335 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967107 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967109 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967113 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967115 4335 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967118 4335 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967121 4335 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967123 4335 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967126 4335 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967129 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967131 4335 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967134 4335 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967137 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967140 4335 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967142 4335 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967145 4335 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:44:26.970990 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967148 4335 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967150 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967153 4335 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967156 4335 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967158 4335 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967161 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967164 4335 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967166 4335 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967169 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967172 4335 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967174 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967177 4335 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967179 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967182 4335 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967184 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967188 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967191 4335 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967193 4335 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967196 4335 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:44:26.971467 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967198 4335 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967201 4335 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967203 4335 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967206 4335 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967208 4335 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967211 4335 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967213 4335 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967216 4335 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967219 4335 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967221 4335 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967226 4335 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967229 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967231 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967235 4335 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967237 4335 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967240 4335 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967242 4335 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967245 4335 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967247 4335 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967250 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:44:26.971985 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967253 4335 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967255 4335 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967258 4335 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967260 4335 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967263 4335 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967265 4335 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967268 4335 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967270 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967274 4335 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967277 4335 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967279 4335 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967282 4335 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967285 4335 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967287 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967290 4335 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967293 4335 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967295 4335 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967298 4335 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967300 4335 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967303 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:44:26.972844 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967307 4335 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967310 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967314 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967316 4335 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967319 4335 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967323 4335 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967327 4335 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967330 4335 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967333 4335 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967335 4335 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967339 4335 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967341 4335 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967344 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967346 4335 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967349 4335 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967351 4335 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967354 4335 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967356 4335 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967360 4335 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:44:26.973677 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967362 4335 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967365 4335 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.967367 4335 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967462 4335 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967473 4335 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967480 4335 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967487 4335 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967493 4335 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967498 4335 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967504 4335 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967510 4335 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967517 4335 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967520 4335 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967523 4335 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967527 4335 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967530 4335 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967533 4335 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967536 4335 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967539 4335 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967542 4335 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967546 4335 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967551 4335 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967554 4335 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967557 4335 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:44:26.974278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967560 4335 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967563 4335 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967567 4335 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967570 4335 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967573 4335 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967576 4335 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967579 4335 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967582 4335 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967586 4335 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967589 4335 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967592 4335 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967596 4335 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967600 4335 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967603 4335 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967606 4335 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967622 4335 flags.go:64] FLAG: --enable-server="true" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967625 4335 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967630 4335 flags.go:64] FLAG: --event-burst="100" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967633 4335 flags.go:64] FLAG: --event-qps="50" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967638 4335 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967641 4335 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967644 4335 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967648 4335 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967651 4335 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967654 4335 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:44:26.974903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967657 4335 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967660 4335 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967663 4335 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967666 4335 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967670 4335 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967673 4335 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967676 4335 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967679 4335 flags.go:64] FLAG: --feature-gates="" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967683 4335 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967686 4335 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967689 4335 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967692 4335 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967695 4335 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967698 4335 flags.go:64] FLAG: --help="false" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967701 4335 flags.go:64] FLAG: --hostname-override="ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967704 4335 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967707 4335 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967710 4335 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967714 4335 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967717 4335 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967720 4335 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967723 4335 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967726 4335 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:44:26.975557 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967729 4335 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967732 4335 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967735 4335 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967740 4335 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967743 4335 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967746 4335 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967749 4335 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967752 4335 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967755 4335 flags.go:64] FLAG: --lock-file="" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967758 4335 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967761 4335 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967764 4335 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967769 4335 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967772 4335 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967776 4335 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967780 4335 flags.go:64] FLAG: --logging-format="text" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967783 4335 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967786 4335 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967790 4335 flags.go:64] FLAG: --manifest-url="" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967792 4335 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967797 4335 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967800 4335 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967804 4335 flags.go:64] FLAG: --max-pods="110" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967807 4335 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967810 4335 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:44:26.976179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967813 4335 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967816 4335 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967819 4335 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967822 4335 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967825 4335 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967832 4335 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967835 4335 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967838 4335 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967841 4335 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967844 4335 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967849 4335 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967852 4335 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967854 4335 flags.go:64] FLAG: --port="10250" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967857 4335 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967860 4335 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-03edbd41cbd6ed55e" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967863 4335 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967866 4335 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967869 4335 flags.go:64] FLAG: --register-node="true" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967872 4335 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967876 4335 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967879 4335 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967882 4335 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967887 4335 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967891 4335 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967894 4335 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967897 4335 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:44:26.976839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967900 4335 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967903 4335 flags.go:64] FLAG: --runonce="false" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967905 4335 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967908 4335 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967911 4335 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967914 4335 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967917 4335 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967920 4335 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967923 4335 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967926 4335 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967929 4335 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967932 4335 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967935 4335 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967938 4335 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967941 4335 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967944 4335 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967948 4335 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967953 4335 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967956 4335 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967959 4335 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967963 4335 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967966 4335 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967969 4335 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967971 4335 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967975 4335 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:44:26.977570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967978 4335 flags.go:64] FLAG: --v="2" Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967982 4335 flags.go:64] FLAG: --version="false" Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967986 4335 flags.go:64] FLAG: --vmodule="" Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967991 4335 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.967995 4335 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968120 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968124 4335 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968127 4335 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968130 4335 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968133 4335 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968137 4335 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968141 4335 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968144 4335 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968147 4335 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968150 4335 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968152 4335 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968155 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968157 4335 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968160 4335 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968163 4335 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:44:26.978329 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968165 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968168 4335 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968170 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968175 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968178 4335 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968180 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968183 4335 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968185 4335 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968188 4335 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968190 4335 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968193 4335 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968196 4335 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968199 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968201 4335 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968204 4335 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968208 4335 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968210 4335 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968213 4335 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968216 4335 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968218 4335 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:44:26.978841 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968221 4335 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968223 4335 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968226 4335 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968229 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968231 4335 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968234 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968236 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968240 4335 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968244 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968247 4335 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968249 4335 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968252 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968255 4335 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968257 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968260 4335 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968264 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968267 4335 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968269 4335 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968272 4335 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:44:26.979358 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968275 4335 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968277 4335 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968280 4335 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968282 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968285 4335 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968287 4335 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968290 4335 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968292 4335 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968296 4335 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968299 4335 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968301 4335 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968304 4335 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968307 4335 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968310 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968312 4335 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968315 4335 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968317 4335 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968320 4335 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968322 4335 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:44:26.979915 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968325 4335 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968327 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968330 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968332 4335 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968335 4335 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968337 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968340 4335 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968342 4335 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968345 4335 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968348 4335 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968351 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968353 4335 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968356 4335 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968358 4335 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968361 4335 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968363 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968366 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968368 4335 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968371 4335 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968373 4335 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:44:26.980464 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968376 4335 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:44:26.981174 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968380 4335 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:44:26.981174 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968383 4335 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:44:26.981174 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968385 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:44:26.981174 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968388 4335 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:44:26.981174 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968391 4335 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:44:26.981174 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968393 4335 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:44:26.981174 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968396 4335 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:44:26.981174 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968399 4335 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:44:26.981174 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968401 4335 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:44:26.981174 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968403 4335 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:44:26.981174 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968406 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:44:26.981174 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968408 4335 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:44:26.981174 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.968433 4335 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:44:26.981174 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968468 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:44:26.981174 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968473 4335 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:44:26.981174 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968476 4335 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968479 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968482 4335 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968484 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968487 4335 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968490 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968494 4335 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968496 4335 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968499 4335 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968502 4335 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968504 4335 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968507 4335 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968510 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968512 4335 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968515 4335 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968518 4335 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968520 4335 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968524 4335 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968526 4335 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:44:26.981630 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968529 4335 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968532 4335 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968534 4335 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968537 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968540 4335 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968542 4335 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968545 4335 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968552 4335 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968555 4335 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968557 4335 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968560 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968562 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968565 4335 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968568 4335 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968570 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968573 4335 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968575 4335 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968578 4335 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968580 4335 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968584 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:44:26.982176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968586 4335 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968589 4335 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968592 4335 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968594 4335 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968597 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968599 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968602 4335 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968604 4335 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968607 4335 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968630 4335 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968636 4335 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968639 4335 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968641 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968644 4335 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968646 4335 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968649 4335 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968652 4335 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968655 4335 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968658 4335 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968660 4335 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:44:26.982918 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968663 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968665 4335 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968668 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968671 4335 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968674 4335 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968676 4335 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968679 4335 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968683 4335 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968686 4335 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968690 4335 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968693 4335 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968697 4335 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968699 4335 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968702 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968705 4335 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968707 4335 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968710 4335 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968712 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968716 4335 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:44:26.983574 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968718 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968721 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968723 4335 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968728 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968730 4335 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968733 4335 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968735 4335 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968738 4335 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968741 4335 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968744 4335 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968747 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968750 4335 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968752 4335 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968755 4335 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968757 4335 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968759 4335 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968762 4335 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968765 4335 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968767 4335 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968770 4335 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:44:26.984112 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968772 4335 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968775 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968778 4335 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968781 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968785 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.968790 4335 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.974861 4335 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.974877 4335 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.974894 4335 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.974899 4335 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.974966 4335 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.974971 4335 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.974975 4335 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.974978 4335 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.974980 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.974983 4335 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.974986 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.974988 4335 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.974991 4335 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:44:26.984604 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.974994 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.974996 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.974999 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975002 4335 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975004 4335 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975007 4335 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975010 4335 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975013 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975016 4335 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975018 4335 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975021 4335 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975025 4335 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975029 4335 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975032 4335 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975035 4335 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975038 4335 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975041 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975044 4335 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975046 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975049 4335 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:44:26.985217 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975053 4335 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975055 4335 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975058 4335 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975060 4335 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975063 4335 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975066 4335 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975069 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975071 4335 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975074 4335 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975076 4335 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975079 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975081 4335 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975084 4335 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975087 4335 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975089 4335 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975092 4335 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975094 4335 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975097 4335 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975100 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975103 4335 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:44:26.986097 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975106 4335 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975109 4335 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975111 4335 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975114 4335 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975117 4335 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975119 4335 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975122 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975125 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975128 4335 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975130 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975133 4335 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975135 4335 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975138 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975140 4335 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975143 4335 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975145 4335 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975148 4335 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975150 4335 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975153 4335 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975155 4335 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:44:26.986924 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975158 4335 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975162 4335 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975165 4335 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975169 4335 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975172 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975175 4335 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975177 4335 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975180 4335 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975182 4335 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975185 4335 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975188 4335 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975191 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975193 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975196 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975198 4335 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975201 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975204 4335 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975206 4335 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975208 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975211 4335 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:44:26.987680 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975214 4335 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:44:26.988556 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975216 4335 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:44:26.988556 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975219 4335 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:44:26.988556 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975221 4335 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:44:26.988556 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975224 4335 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:44:26.988556 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975227 4335 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:44:26.988556 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975229 4335 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:44:26.988556 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975232 4335 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:44:26.988556 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975234 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:44:26.988556 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975237 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:44:26.988556 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975240 4335 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:44:26.988556 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975243 4335 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:44:26.988556 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975245 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:44:26.988556 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975248 4335 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:44:26.988556 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975250 4335 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:44:26.988556 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975253 4335 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:44:26.988556 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975255 4335 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:44:26.989339 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.975283 4335 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:44:26.989339 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975424 4335 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:44:26.989339 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975428 4335 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:44:26.989339 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975432 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:44:26.989339 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975436 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:44:26.989339 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975439 4335 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:44:26.989339 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975442 4335 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:44:26.989339 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975445 4335 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:44:26.989339 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975447 4335 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:44:26.989339 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975450 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:44:26.989339 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975453 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:44:26.989339 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975455 4335 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:44:26.989339 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975457 4335 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:44:26.989339 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975460 4335 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:44:26.989339 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975462 4335 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:44:26.989339 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975465 4335 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975468 4335 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975471 4335 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975473 4335 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975476 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975478 4335 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975481 4335 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975484 4335 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975488 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975491 4335 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975493 4335 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975496 4335 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975499 4335 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975501 4335 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975504 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975506 4335 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975509 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975512 4335 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975515 4335 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:44:26.989883 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975517 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975520 4335 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975523 4335 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975525 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975528 4335 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975531 4335 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975534 4335 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975536 4335 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975539 4335 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975541 4335 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975544 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975546 4335 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975549 4335 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975552 4335 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975554 4335 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975557 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975560 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975562 4335 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975565 4335 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975568 4335 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:44:26.990419 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975570 4335 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975573 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975576 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975578 4335 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975581 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975583 4335 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975586 4335 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975588 4335 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975592 4335 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975596 4335 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975599 4335 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975602 4335 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975606 4335 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975609 4335 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975627 4335 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975630 4335 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975632 4335 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975635 4335 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975637 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:44:26.990929 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975640 4335 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975643 4335 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975645 4335 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975648 4335 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975650 4335 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975653 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975655 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975658 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975661 4335 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975663 4335 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975666 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975668 4335 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975671 4335 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975674 4335 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975676 4335 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975679 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975681 4335 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975684 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975686 4335 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975688 4335 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:44:26.991408 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975691 4335 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:44:26.991950 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975694 4335 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:44:26.991950 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975696 4335 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:44:26.991950 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975699 4335 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:44:26.991950 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975701 4335 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:44:26.991950 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975704 4335 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:44:26.991950 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975707 4335 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:44:26.991950 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975709 4335 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:44:26.991950 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975712 4335 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:44:26.991950 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975714 4335 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:44:26.991950 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975717 4335 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:44:26.991950 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975719 4335 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:44:26.991950 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975723 4335 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:44:26.991950 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975725 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:44:26.991950 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.975749 4335 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:44:26.991950 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975800 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:44:26.991950 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975804 4335 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975807 4335 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975810 4335 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975813 4335 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975817 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975819 4335 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975822 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975824 4335 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975827 4335 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975830 4335 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975832 4335 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975835 4335 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975837 4335 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975840 4335 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975842 4335 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975845 4335 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975847 4335 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975850 4335 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975852 4335 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975855 4335 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:44:26.992333 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975857 4335 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975860 4335 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975862 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975865 4335 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975867 4335 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975870 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975872 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975874 4335 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975877 4335 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975880 4335 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975882 4335 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975885 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975888 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975890 4335 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975892 4335 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975895 4335 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975898 4335 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975900 4335 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975902 4335 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:44:26.992901 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975905 4335 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975907 4335 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975910 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975912 4335 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975915 4335 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975919 4335 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975923 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975930 4335 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975933 4335 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975936 4335 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975939 4335 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975942 4335 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975945 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975948 4335 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975950 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975953 4335 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975955 4335 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975958 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975961 4335 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975965 4335 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:44:26.993375 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975967 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975970 4335 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975972 4335 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975975 4335 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975978 4335 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975981 4335 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975983 4335 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975986 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975989 4335 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975991 4335 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975994 4335 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975996 4335 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.975999 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976001 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976004 4335 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976006 4335 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976009 4335 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976011 4335 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976014 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976018 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:44:26.993874 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976020 4335 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976023 4335 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976025 4335 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976028 4335 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976031 4335 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976033 4335 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976037 4335 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976040 4335 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976043 4335 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976046 4335 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976049 4335 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976052 4335 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976054 4335 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976057 4335 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976059 4335 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976062 4335 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976066 4335 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976069 4335 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976071 4335 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:44:26.994396 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976074 4335 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:44:26.994893 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976076 4335 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:44:26.994893 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976079 4335 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:44:26.994893 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976082 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:44:26.994893 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976084 4335 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:44:26.994893 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976088 4335 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:44:26.994893 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:26.976090 4335 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:44:26.994893 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.976141 4335 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:44:26.994893 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.977219 4335 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:44:26.994893 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.977282 4335 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:44:26.994893 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.977524 4335 server.go:1014] "Starting client certificate rotation" Oct 30 21:44:26.994893 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.977661 4335 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:44:26.994893 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.977739 4335 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:38:47 +0000 UTC" deadline="2028-04-27 20:53:08.54882184 +0000 UTC" Oct 30 21:44:26.994893 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.977788 4335 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13079h8m41.571037259s" Oct 30 21:44:26.994893 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.978092 4335 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:44:26.994893 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.979089 4335 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:44:26.994893 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.980759 4335 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:44:26.995278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.982434 4335 log.go:25] "Validated CRI v1 image API" Oct 30 21:44:26.995278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.983251 4335 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:44:26.995278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.985290 4335 fs.go:135] Filesystem UUIDs: map[74dfd489-63be-45b2-9267-ff925c629e5c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d48cc798-0955-4ba8-802c-943780d36880:/dev/nvme0n1p4] Oct 30 21:44:26.995362 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.985318 4335 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03516dcf44f10b5c29eaccb99085929e2c26bbe9b1dd584f83a2034331c977fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03516dcf44f10b5c29eaccb99085929e2c26bbe9b1dd584f83a2034331c977fb/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/052ec105874a388a6268608054e30666c883b13aef72d2e9ab2de4d7b5da36fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/052ec105874a388a6268608054e30666c883b13aef72d2e9ab2de4d7b5da36fd/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/196cde0e820e2f4d7dfd3a3316445ad7a192fd2fb3f5559fe367a5d575cd2956/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/196cde0e820e2f4d7dfd3a3316445ad7a192fd2fb3f5559fe367a5d575cd2956/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37442a05753618fe8f9a34c180b2490a39fb6babbff7a4e0414d23e01cd90209/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37442a05753618fe8f9a34c180b2490a39fb6babbff7a4e0414d23e01cd90209/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d1b6a0958405e70a9f74d458070972d7112ad956c090e4ac078281e0474474a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d1b6a0958405e70a9f74d458070972d7112ad956c090e4ac078281e0474474a/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55c67840b3a6b6d25f6059d282483d6436754966dd76cadfbab8fba03c0c5f16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55c67840b3a6b6d25f6059d282483d6436754966dd76cadfbab8fba03c0c5f16/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7450f996eafba6511069b0b4bd3c37035e90292311482e39304f970f045b75c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7450f996eafba6511069b0b4bd3c37035e90292311482e39304f970f045b75c6/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7be60c6591fc4f959088b7fe0725ed1b1749b5bb4d768ef330aa5e72d0671a76/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7be60c6591fc4f959088b7fe0725ed1b1749b5bb4d768ef330aa5e72d0671a76/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab84e6c2898e89b5c29be5740b8bde81a420d3c21761cd0804d7fa890f1c4bfd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab84e6c2898e89b5c29be5740b8bde81a420d3c21761cd0804d7fa890f1c4bfd/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afa919c15cf04a00e9c88be0ab38c60002f01b4d0fa843a67ceeb4f034748f85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afa919c15cf04a00e9c88be0ab38c60002f01b4d0fa843a67ceeb4f034748f85/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b151e7777a44f9a184b6ba2b452dbd24cf00f56bb074af9d289768464a80be0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b151e7777a44f9a184b6ba2b452dbd24cf00f56bb074af9d289768464a80be0e/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf792177a2b8333824e37fc3a6ada83da994aa5a4ed790617bb8bba444ad52c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf792177a2b8333824e37fc3a6ada83da994aa5a4ed790617bb8bba444ad52c6/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc8f0960d9708f19011021a19fbce978ac0df060b9f68e9b9086b32cea43803b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc8f0960d9708f19011021a19fbce978ac0df060b9f68e9b9086b32cea43803b/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb4d4ca5723228ea9975e335fa64c10fe04d537c57c0dbe7f87209cbc3b2c958/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb4d4ca5723228ea9975e335fa64c10fe04d537c57c0dbe7f87209cbc3b2c958/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8d8f5bd04a7a184c288d5eda923787389f21397007a832e3d324e56377077d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8d8f5bd04a7a184c288d5eda923787389f21397007a832e3d324e56377077d0/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff888be3d101a97039d7b7ff621081a65a1d5bf86559c62342cc75095a572cfb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff888be3d101a97039d7b7ff621081a65a1d5bf86559c62342cc75095a572cfb/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/000931e4-5bd3-4387-adf7-ea47151527ad/volumes/kubernetes.io~projected/kube-api-access-9gv95:{mountpoint:/var/lib/kubelet/pods/000931e4-5bd3-4387-adf7-ea47151527ad/volumes/kubernetes.io~projected/kube-api-access-9gv95 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08685d97-2f18-4abe-a7b8-25466aee77cb/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/08685d97-2f18-4abe-a7b8-25466aee77cb/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08685d97-2f18-4abe-a7b8-25466aee77cb/volumes/kubernetes.io~projected/kube-api-access-hbn2h:{mountpoint:/var/lib/kubelet/pods/08685d97-2f18-4abe-a7b8-25466aee77cb/volumes/kubernetes.io~projected/kube-api-access-hbn2h major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08685d97-2f18-4abe-a7b8-25466aee77cb/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/08685d97-2f18-4abe-a7b8-25466aee77cb/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08df8d3b-f392-438a-81d5-3ee04ee2cd8b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/08df8d3b-f392-438a-81d5-3ee04ee2cd8b/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12b346ba-5a09-4ec5-8c16-885fbe1502bc/volumes/kubernetes.io~projected/kube-api-access-pq6b2:{mountpoint:/var/lib/kubelet/pods/12b346ba-5a09-4ec5-8c16-885fbe1502bc/volumes/kubernetes.io~projected/kube-api-access-pq6b2 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12b346ba-5a09-4ec5-8c16-885fbe1502bc/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/12b346ba-5a09-4ec5-8c16-885fbe1502bc/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432/volumes/kubernetes.io~projected/kube-api-access-4wfjl:{mountpoint:/var/lib/kubelet/pods/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432/volumes/kubernetes.io~projected/kube-api-access-4wfjl major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a899495-aa25-4538-8eef-5640503b466a/volumes/kubernetes.io~projected/kube-api-access-p8l9c:{mountpoint:/var/lib/kubelet/pods/2a899495-aa25-4538-8eef-5640503b466a/volumes/kubernetes.io~projected/kube-api-access-p8l9c major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a846703-92eb-4115-875f-c347fc4f2a1c/volumes/kubernetes.io~projected/kube-api-access-tvscc:{mountpoint:/var/lib/kubelet/pods/3a846703-92eb-4115-875f-c347fc4f2a1c/volumes/kubernetes.io~projected/kube-api-access-tvscc major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ee9e6d0-f122-4ff3-aa72-b597339928ac/volumes/kubernetes.io~projected/kube-api-access-5cgbv:{mountpoint:/var/lib/kubelet/pods/3ee9e6d0-f122-4ff3-aa72-b597339928ac/volumes/kubernetes.io~projected/kube-api-access-5cgbv major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535/volumes/kubernetes.io~projected/kube-api-access-hjtfp:{mountpoint:/var/lib/kubelet/pods/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535/volumes/kubernetes.io~projected/kube-api-access-hjtfp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74fe5a17-9648-48a9-8d1b-338990dac55b/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/74fe5a17-9648-48a9-8d1b-338990dac55b/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74fe5a17-9648-48a9-8d1b-338990dac55b/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/74fe5a17-9648-48a9-8d1b-338990dac55b/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74fe5a17-9648-48a9-8d1b-338990dac55b/volumes/kubernetes.io~projected/kube-api-access-46rpn:{mountpoint:/var/lib/kubelet/pods/74fe5a17-9648-48a9-8d1b-338990dac55b/volumes/kubernetes.io~projected/kube-api-access-46rpn major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/827f9fb3-0eb3-4f8f-96f2-72df80c024d8/volumes/kubernetes.io~projected/kube-api-access-v9qk6:{mountpoint:/var/lib/kubelet/pods/827f9fb3-0eb3-4f8f-96f2-72df80c024d8/volumes/kubernetes.io~projected/kube-api-access-v9qk6 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f03953c-1f9f-4b43-8f1d-0a5f3766340b/volumes/kubernetes.io~projected/kube-api-access-64p8r:{mountpoint:/var/lib/kubelet/pods/8f03953c-1f9f-4b43-8f1d-0a5f3766340b/volumes/kubernetes.io~projected/kube-api-access-64p8r major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f03953c-1f9f-4b43-8f1d-0a5f3766340b/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/8f03953c-1f9f-4b43-8f1d-0a5f3766340b/volumes/kubernetes.io~secret/signing-key major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92da1846-1753-43a7-84a7-3f317e84675d/volumes/kubernetes.io~projected/kube-api-access-m7k9p:{mountpoint:/var/lib/kubelet/pods/92da1846-1753-43a7-84a7-3f317e84675d/volumes/kubernetes.io~projected/kube-api-access-m7k9p major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c94b6070-d303-4c76-8895-73cd37a2f985/volumes/kubernetes.io~projected/kube-api-access-bzbrt:{mountpoint:/var/lib/kubelet/pods/c94b6070-d303-4c76-8895-73cd37a2f985/volumes/kubernetes.io~projected/kube-api-access-bzbrt major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d45916e8-c498-4bf5-b5f3-df697b608ed9/volumes/kubernetes.io~projected/kube-api-access-lrgmx:{mountpoint:/var/lib/kubelet/pods/d45916e8-c498-4bf5-b5f3-df697b608ed9/volumes/kubernetes.io~projected/kube-api-access-lrgmx major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1668e0e-55e0-493c-a735-1ff0da66eda3/volumes/kubernetes.io~projected/kube-api-access-5cbbb:{mountpoint:/var/lib/kubelet/pods/e1668e0e-55e0-493c-a735-1ff0da66eda3/volumes/kubernetes.io~projected/kube-api-access-5cbbb major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee354605-03c9-4c69-a937-5f94440dc0ae/volumes/kubernetes.io~projected/kube-api-access-7nt72:{mountpoint:/var/lib/kubelet/pods/ee354605-03c9-4c69-a937-5f94440dc0ae/volumes/kubernetes.io~projected/kube-api-access-7nt72 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f73329a3-7290-44a1-a72e-1fe72c0d0da5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f73329a3-7290-44a1-a72e-1fe72c0d0da5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:212 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/be280bfbbde4b04a0c9db63f39aff2bb7872d04d9023458ade82b8395eef2fc5/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/15ea30e855dd933e9331979f2297dd47fd100c6725d2afa4e208178285626a86/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c4ebb75f7a1ef82c9decd7e6d14429c52b1b9aad3016a5bd934d2b82e0c4a67b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/d71b946eb2c8dd61341bb3f812b6665f960b3978cc66efb0bbe9f85866ff844f/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/8936bd34331db93071be497fe548bb55baadf17a54d7cf5528241689f45394f7/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/cd98d373452e4f1289fc9e7ba89a8fdf1652e7e330143b6632575f596586c4fa/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-121:{mountpoint:/var/lib/containers/storage/overlay/a09241219ce78788fce936354de73925f9bb7b41abdf3fb00acffc5c1afecdc5/merged major:0 minor:121 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/0d46a911d01257ec7225f284fcf2481ccb841d0495883939f50e56501e2090b4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/a68acb042d1f052eb7756fa93c2780007faf1d8b69589ed4bf0435504843d03d/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/345df8e131dbf7713c183b1ad00a10ea0a9d3c3847dae4f58703316653bb3168/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/c015f80a93aa82b22ea29dccfa7ebe67683c54754d6fa91a249d7ea2d60794d2/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/e9e44c42b254e996c8fbdc2b854d4cda19f5ea6fe2652e1d3cd7a0db9c413295/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/9902bed1629687c7755acfec5527031e24a18b400be414c4831aa3c1b068a7f6/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/7339b25ea7b078c709bfab3eb93e1392f37b751d965782548787c39aea8182c1/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/869115e0460610c6bf032fd29c896009a1c2baf2a8b8ff9b8f5f8d5d39b407d4/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/f1e733bc21f46f279f8f23d8365d1f727e609af9d3bd60f7234f7a38b0b39dbd/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/aadd6f9ab379b6ffca621c4dde2fdfb0f8224b893d6bfd6afa510c7ab9b8d37a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/38ab71ccad9d6838a3d4874bd5855e7346a39a2ed7df461b06c3e981513d9289/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/b23be0518024bf7ee85ffbdbe488ec8293101fc87030bf5489e79c794d86daab/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f44d3cfbfdb6b65e9dad0abf48d36c70d4ef451eaaf66574ebeb162a25f283db/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/88d021a423ce8c9d15df58956fc1e120902d2c6257b8c6845331ee3aab61ee33/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/0883a5525b26a27897aafc7b2b6313bac8f7b43c4984cbaa1437f3b1e808aa2b/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/53743f2eded80a265ae3462e183a1b588b178ddfe01a97b02d4db70736cd9101/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/613547a92672e7404782fe7df0694d7fc2bc4e522c7bd24c64cbf71f31272f71/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/26ccf3dbce3764e799046aeb0f33218ea820c2672ccb762daf3faf82928fcba8/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/b51e9c118ee13aaf696dcd0779315db4df17d9199d1e692f59e36bd92af27dc1/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/5763526995f82803fd27702b41c60d16806bf7be80da2d148bde005586625fc7/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/865d768346c18d46c2d5ff60ab25bc41a5265dbd706c0f83f8dd2bd19cdfce78/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/cb4afa75834e2d4493bbf368a7c57896c9a15c8f8d1d3254288d0062801d948e/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/49c16241ae67137f8e3ef5188d693e93bdb5c47f4c110824157635b99b26587b/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/46351f5bcd6a5c05b4594631ae3b017281a7ed37cb7ede8ce219ad2bafa428cc/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/78f0eb994a1058452d9e8f8a55f72ff2ab9ab0d1f503b051378d18fce7931290/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/2f608950c4da5fd0d5df8314a72f700a91235d5395d2b3334695c1ece6b3d02c/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/da2020e0725f05ea6a8b3b2633054bfe818f587c21811583f35e7200d89ec04f/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/353250fce765907bb6d8f4e3636f517f3d02748a65d0cf4ac4ed995eaa9241fa/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/662b9b495b72d5a1283ae46287f6f1e7489342cc614698c4facaefbef17f46e8/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/4ecc57742cb9e96b90ee4713758db17893985d898dc95795dda0d4c91a3f9804/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/70b3349deaf4886bb0f7c9d9f8cb574e3bb409e5842d3f1d9df1c6565b489dc3/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/27b4fa6e01e91660b81800894db5da61fdaa421ab4c49fd61f919fadc0ddfb66/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/af36a7a0502c4caaf19a4f8c86e55d07f18e02a41017c86f7de4903178d7829b/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/f04b7b9b9d7141585f5932c5f62f03cdc789d76885b5fa7a933cead1e0f46152/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/7e550498488c253eadfb6ad8506be9970147a27768977a56183a9ef1065a78b3/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/11fde21e5b536f2bc365ffe821b0b566d6d8a320c14ec308701ac34b6f8825df/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 21:44:26.995608 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.995355 4335 manager.go:217] Machine: {Timestamp:2026-10-30 21:44:26.994340207 +0000 UTC m=+0.094771936 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095759 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23808bec0d619ad363015ef799f5d5 SystemUUID:ec23808b-ec0d-619a-d363-015ef799f5d5 BootID:18424bbf-1c5f-42fc-bc34-9ff446645789 Filesystems:[{Device:/var/lib/kubelet/pods/08685d97-2f18-4abe-a7b8-25466aee77cb/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff888be3d101a97039d7b7ff621081a65a1d5bf86559c62342cc75095a572cfb/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ee9e6d0-f122-4ff3-aa72-b597339928ac/volumes/kubernetes.io~projected/kube-api-access-5cgbv DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2a899495-aa25-4538-8eef-5640503b466a/volumes/kubernetes.io~projected/kube-api-access-p8l9c DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08685d97-2f18-4abe-a7b8-25466aee77cb/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7450f996eafba6511069b0b4bd3c37035e90292311482e39304f970f045b75c6/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/92da1846-1753-43a7-84a7-3f317e84675d/volumes/kubernetes.io~projected/kube-api-access-m7k9p DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55c67840b3a6b6d25f6059d282483d6436754966dd76cadfbab8fba03c0c5f16/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74fe5a17-9648-48a9-8d1b-338990dac55b/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432/volumes/kubernetes.io~projected/kube-api-access-4wfjl DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535/volumes/kubernetes.io~projected/kube-api-access-hjtfp DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d1b6a0958405e70a9f74d458070972d7112ad956c090e4ac078281e0474474a/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf792177a2b8333824e37fc3a6ada83da994aa5a4ed790617bb8bba444ad52c6/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08df8d3b-f392-438a-81d5-3ee04ee2cd8b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c94b6070-d303-4c76-8895-73cd37a2f985/volumes/kubernetes.io~projected/kube-api-access-bzbrt DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/12b346ba-5a09-4ec5-8c16-885fbe1502bc/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/000931e4-5bd3-4387-adf7-ea47151527ad/volumes/kubernetes.io~projected/kube-api-access-9gv95 DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74fe5a17-9648-48a9-8d1b-338990dac55b/volumes/kubernetes.io~projected/kube-api-access-46rpn DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03516dcf44f10b5c29eaccb99085929e2c26bbe9b1dd584f83a2034331c977fb/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a846703-92eb-4115-875f-c347fc4f2a1c/volumes/kubernetes.io~projected/kube-api-access-tvscc DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8d8f5bd04a7a184c288d5eda923787389f21397007a832e3d324e56377077d0/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb4d4ca5723228ea9975e335fa64c10fe04d537c57c0dbe7f87209cbc3b2c958/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afa919c15cf04a00e9c88be0ab38c60002f01b4d0fa843a67ceeb4f034748f85/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc8f0960d9708f19011021a19fbce978ac0df060b9f68e9b9086b32cea43803b/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b151e7777a44f9a184b6ba2b452dbd24cf00f56bb074af9d289768464a80be0e/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f03953c-1f9f-4b43-8f1d-0a5f3766340b/volumes/kubernetes.io~projected/kube-api-access-64p8r DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/74fe5a17-9648-48a9-8d1b-338990dac55b/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ee354605-03c9-4c69-a937-5f94440dc0ae/volumes/kubernetes.io~projected/kube-api-access-7nt72 DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-121 DeviceMajor:0 DeviceMinor:121 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/e1668e0e-55e0-493c-a735-1ff0da66eda3/volumes/kubernetes.io~projected/kube-api-access-5cbbb DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab84e6c2898e89b5c29be5740b8bde81a420d3c21761cd0804d7fa890f1c4bfd/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d45916e8-c498-4bf5-b5f3-df697b608ed9/volumes/kubernetes.io~projected/kube-api-access-lrgmx DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8f03953c-1f9f-4b43-8f1d-0a5f3766340b/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/827f9fb3-0eb3-4f8f-96f2-72df80c024d8/volumes/kubernetes.io~projected/kube-api-access-v9qk6 DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/08685d97-2f18-4abe-a7b8-25466aee77cb/volumes/kubernetes.io~projected/kube-api-access-hbn2h DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37442a05753618fe8f9a34c180b2490a39fb6babbff7a4e0414d23e01cd90209/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12b346ba-5a09-4ec5-8c16-885fbe1502bc/volumes/kubernetes.io~projected/kube-api-access-pq6b2 DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f73329a3-7290-44a1-a72e-1fe72c0d0da5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/196cde0e820e2f4d7dfd3a3316445ad7a192fd2fb3f5559fe367a5d575cd2956/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/052ec105874a388a6268608054e30666c883b13aef72d2e9ab2de4d7b5da36fd/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7be60c6591fc4f959088b7fe0725ed1b1749b5bb4d768ef330aa5e72d0671a76/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:052ec105874a388 MacAddress:da:5d:58:eb:91:de Speed:10000 Mtu:8901} {Name:196cde0e820e2f4 MacAddress:42:97:ab:79:10:36 Speed:10000 Mtu:8901} {Name:55c67840b3a6b6d MacAddress:0e:30:a7:ba:d9:87 Speed:10000 Mtu:8901} {Name:7be60c6591fc4f9 MacAddress:ce:97:cb:a5:7f:b2 Speed:10000 Mtu:8901} {Name:b151e7777a44f9a MacAddress:e2:c2:8d:1b:eb:4c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:4b:9f:f8:63:13 Speed:0 Mtu:9001} {Name:br-int MacAddress:c2:4a:e6:92:ff:84 Speed:0 Mtu:8901} {Name:eb4d4ca5723228e MacAddress:e6:14:7f:6b:2f:07 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:4b:9f:f8:63:13 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:d2:55:c8:a1:c1:f9 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:96:27:19:f0:bb:5e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:44:26.995608 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.995636 4335 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:44:26.995852 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.995741 4335 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:44:26.996040 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.996022 4335 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:44:26.996157 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.996041 4335 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-193.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:44:26.996201 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.996159 4335 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:44:26.996201 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.996167 4335 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:44:26.996201 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.996177 4335 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:44:26.996201 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.996190 4335 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:44:26.996298 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.996241 4335 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:44:26.996391 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.996381 4335 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:44:26.996492 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.996484 4335 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:44:26.996530 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.996521 4335 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:44:26.996577 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.996538 4335 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:44:26.996577 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.996558 4335 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:44:26.996577 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.996572 4335 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:44:26.996743 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.996583 4335 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:44:26.997312 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.997298 4335 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:44:26.997349 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.997316 4335 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:44:26.997852 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.997831 4335 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:44:26.999932 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.999916 4335 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:44:26.999932 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:26.999932 4335 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:44:27.000049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.000030 4335 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:44:27.000210 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.000199 4335 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:44:27.000239 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.000228 4335 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:44:27.000311 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.000302 4335 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:44:27.000351 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.000318 4335 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:44:27.000351 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.000324 4335 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:44:27.000351 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.000329 4335 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:44:27.000351 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.000334 4335 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:44:27.000351 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.000341 4335 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:44:27.000351 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.000346 4335 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:44:27.000351 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.000353 4335 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:44:27.000601 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.000360 4335 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:44:27.000601 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.000371 4335 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:44:27.000601 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.000387 4335 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:44:27.000601 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.000397 4335 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:44:27.000601 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.000422 4335 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:44:27.000601 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.000427 4335 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:44:27.004897 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.004202 4335 server.go:1263] "Started kubelet" Oct 30 21:44:27.005655 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.004226 4335 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:44:27.006249 ip-10-0-131-193 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:44:27.011794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.004334 4335 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:44:27.011794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.006806 4335 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:44:27.011794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.007146 4335 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:44:27.011794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.008239 4335 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:44:27.011794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.008779 4335 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:44:27.011794 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:44:27.009160 4335 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-193.us-west-1.compute.internal\" not found" Oct 30 21:44:27.011794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.009601 4335 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:44:27.011794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.009690 4335 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:38:48 +0000 UTC" deadline="2028-06-04 02:37:48.736160048 +0000 UTC" Oct 30 21:44:27.011794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.009720 4335 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13972h53m21.726443509s" Oct 30 21:44:27.011794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.009815 4335 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:44:27.014342 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.012596 4335 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:44:27.014342 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.012656 4335 factory.go:55] Registering systemd factory Oct 30 21:44:27.014342 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.012667 4335 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:44:27.014342 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.009267 4335 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:44:27.014342 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.012694 4335 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:44:27.017140 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017081 4335 factory.go:153] Registering CRI-O factory Oct 30 21:44:27.017140 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017100 4335 factory.go:223] Registration of the crio container factory successfully Oct 30 21:44:27.017140 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017126 4335 factory.go:103] Registering Raw factory Oct 30 21:44:27.017140 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017140 4335 manager.go:1199] Started watching for new ooms in manager Oct 30 21:44:27.017374 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017284 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="000931e4-5bd3-4387-adf7-ea47151527ad" volumeName="kubernetes.io/configmap/000931e4-5bd3-4387-adf7-ea47151527ad-config-volume" seLinuxMountContext="" Oct 30 21:44:27.017374 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017314 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08685d97-2f18-4abe-a7b8-25466aee77cb" volumeName="kubernetes.io/configmap/08685d97-2f18-4abe-a7b8-25466aee77cb-env-overrides" seLinuxMountContext="" Oct 30 21:44:27.017374 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017329 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a899495-aa25-4538-8eef-5640503b466a" volumeName="kubernetes.io/configmap/2a899495-aa25-4538-8eef-5640503b466a-multus-daemon-config" seLinuxMountContext="" Oct 30 21:44:27.017374 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017343 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ee9e6d0-f122-4ff3-aa72-b597339928ac" volumeName="kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv" seLinuxMountContext="" Oct 30 21:44:27.017374 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017357 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d45916e8-c498-4bf5-b5f3-df697b608ed9" volumeName="kubernetes.io/projected/d45916e8-c498-4bf5-b5f3-df697b608ed9-kube-api-access-lrgmx" seLinuxMountContext="" Oct 30 21:44:27.017374 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017370 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="000931e4-5bd3-4387-adf7-ea47151527ad" volumeName="kubernetes.io/empty-dir/000931e4-5bd3-4387-adf7-ea47151527ad-tmp-dir" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017383 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08685d97-2f18-4abe-a7b8-25466aee77cb" volumeName="kubernetes.io/projected/08685d97-2f18-4abe-a7b8-25466aee77cb-kube-api-access-hbn2h" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017396 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535" volumeName="kubernetes.io/projected/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535-kube-api-access-hjtfp" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017412 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c94b6070-d303-4c76-8895-73cd37a2f985" volumeName="kubernetes.io/projected/c94b6070-d303-4c76-8895-73cd37a2f985-kube-api-access-bzbrt" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017446 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d45916e8-c498-4bf5-b5f3-df697b608ed9" volumeName="kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-service-ca-bundle" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017460 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d45916e8-c498-4bf5-b5f3-df697b608ed9" volumeName="kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017469 4335 manager.go:319] Starting recovery of all containers Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017472 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12b346ba-5a09-4ec5-8c16-885fbe1502bc" volumeName="kubernetes.io/projected/12b346ba-5a09-4ec5-8c16-885fbe1502bc-kube-api-access-pq6b2" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017486 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ac3165a-2ba0-4be9-8ab9-4d088d9a1432" volumeName="kubernetes.io/projected/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-kube-api-access-4wfjl" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017498 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74fe5a17-9648-48a9-8d1b-338990dac55b" volumeName="kubernetes.io/empty-dir/74fe5a17-9648-48a9-8d1b-338990dac55b-tmp" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017514 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f03953c-1f9f-4b43-8f1d-0a5f3766340b" volumeName="kubernetes.io/configmap/8f03953c-1f9f-4b43-8f1d-0a5f3766340b-signing-cabundle" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017539 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f73329a3-7290-44a1-a72e-1fe72c0d0da5" volumeName="kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017551 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08685d97-2f18-4abe-a7b8-25466aee77cb" volumeName="kubernetes.io/configmap/08685d97-2f18-4abe-a7b8-25466aee77cb-ovnkube-config" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017562 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12b346ba-5a09-4ec5-8c16-885fbe1502bc" volumeName="kubernetes.io/secret/12b346ba-5a09-4ec5-8c16-885fbe1502bc-cert" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017573 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535" volumeName="kubernetes.io/configmap/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017586 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74fe5a17-9648-48a9-8d1b-338990dac55b" volumeName="kubernetes.io/empty-dir/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-tuned" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017597 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74fe5a17-9648-48a9-8d1b-338990dac55b" volumeName="kubernetes.io/projected/74fe5a17-9648-48a9-8d1b-338990dac55b-kube-api-access-46rpn" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017608 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f03953c-1f9f-4b43-8f1d-0a5f3766340b" volumeName="kubernetes.io/secret/8f03953c-1f9f-4b43-8f1d-0a5f3766340b-signing-key" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017654 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92da1846-1753-43a7-84a7-3f317e84675d" volumeName="kubernetes.io/projected/92da1846-1753-43a7-84a7-3f317e84675d-kube-api-access-m7k9p" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017667 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee354605-03c9-4c69-a937-5f94440dc0ae" volumeName="kubernetes.io/configmap/ee354605-03c9-4c69-a937-5f94440dc0ae-serviceca" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017678 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="000931e4-5bd3-4387-adf7-ea47151527ad" volumeName="kubernetes.io/projected/000931e4-5bd3-4387-adf7-ea47151527ad-kube-api-access-9gv95" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017705 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08df8d3b-f392-438a-81d5-3ee04ee2cd8b" volumeName="kubernetes.io/configmap/08df8d3b-f392-438a-81d5-3ee04ee2cd8b-konnectivity-ca" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017716 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08df8d3b-f392-438a-81d5-3ee04ee2cd8b" volumeName="kubernetes.io/secret/08df8d3b-f392-438a-81d5-3ee04ee2cd8b-agent-certs" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017730 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a899495-aa25-4538-8eef-5640503b466a" volumeName="kubernetes.io/projected/2a899495-aa25-4538-8eef-5640503b466a-kube-api-access-p8l9c" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017748 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="827f9fb3-0eb3-4f8f-96f2-72df80c024d8" volumeName="kubernetes.io/empty-dir/827f9fb3-0eb3-4f8f-96f2-72df80c024d8-tmp-dir" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017759 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d45916e8-c498-4bf5-b5f3-df697b608ed9" volumeName="kubernetes.io/empty-dir/d45916e8-c498-4bf5-b5f3-df697b608ed9-archives-path" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017771 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee354605-03c9-4c69-a937-5f94440dc0ae" volumeName="kubernetes.io/projected/ee354605-03c9-4c69-a937-5f94440dc0ae-kube-api-access-7nt72" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017797 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08685d97-2f18-4abe-a7b8-25466aee77cb" volumeName="kubernetes.io/configmap/08685d97-2f18-4abe-a7b8-25466aee77cb-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017811 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08685d97-2f18-4abe-a7b8-25466aee77cb" volumeName="kubernetes.io/secret/08685d97-2f18-4abe-a7b8-25466aee77cb-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017823 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ac3165a-2ba0-4be9-8ab9-4d088d9a1432" volumeName="kubernetes.io/configmap/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-cni-binary-copy" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017847 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ac3165a-2ba0-4be9-8ab9-4d088d9a1432" volumeName="kubernetes.io/configmap/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017861 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a846703-92eb-4115-875f-c347fc4f2a1c" volumeName="kubernetes.io/projected/3a846703-92eb-4115-875f-c347fc4f2a1c-kube-api-access-tvscc" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017874 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f03953c-1f9f-4b43-8f1d-0a5f3766340b" volumeName="kubernetes.io/projected/8f03953c-1f9f-4b43-8f1d-0a5f3766340b-kube-api-access-64p8r" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017885 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92da1846-1753-43a7-84a7-3f317e84675d" volumeName="kubernetes.io/empty-dir/92da1846-1753-43a7-84a7-3f317e84675d-data-volume" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017907 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1668e0e-55e0-493c-a735-1ff0da66eda3" volumeName="kubernetes.io/projected/e1668e0e-55e0-493c-a735-1ff0da66eda3-kube-api-access-5cbbb" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017921 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ac3165a-2ba0-4be9-8ab9-4d088d9a1432" volumeName="kubernetes.io/configmap/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017931 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a899495-aa25-4538-8eef-5640503b466a" volumeName="kubernetes.io/configmap/2a899495-aa25-4538-8eef-5640503b466a-cni-binary-copy" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017942 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="827f9fb3-0eb3-4f8f-96f2-72df80c024d8" volumeName="kubernetes.io/projected/827f9fb3-0eb3-4f8f-96f2-72df80c024d8-kube-api-access-v9qk6" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017955 4335 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92da1846-1753-43a7-84a7-3f317e84675d" volumeName="kubernetes.io/configmap/92da1846-1753-43a7-84a7-3f317e84675d-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017966 4335 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:44:27.018855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.017974 4335 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:44:27.028731 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.028558 4335 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:44:27.049324 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.049296 4335 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:44:27.049424 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.049330 4335 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:44:27.049424 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.049350 4335 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:44:27.049424 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:44:27.049387 4335 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:44:27.110665 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:44:27.109674 4335 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-193.us-west-1.compute.internal\" not found" Oct 30 21:44:27.115244 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.114876 4335 manager.go:324] Recovery completed Oct 30 21:44:27.126098 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.126071 4335 generic.go:356] "Generic (PLEG): container finished" podID="1ac3165a-2ba0-4be9-8ab9-4d088d9a1432" containerID="ad3c2cc0ac07ab587124ca5f618ae9c52fa9dc9ea5c3e858b62913b62fb353a3" exitCode=0 Oct 30 21:44:27.126098 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.126095 4335 generic.go:356] "Generic (PLEG): container finished" podID="1ac3165a-2ba0-4be9-8ab9-4d088d9a1432" containerID="95cd1c0f14efee7c6b788e4f8a3033ea3ec5ddc09cdb7c21d4748edbcc8a2e2a" exitCode=0 Oct 30 21:44:27.126098 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.126102 4335 generic.go:356] "Generic (PLEG): container finished" podID="1ac3165a-2ba0-4be9-8ab9-4d088d9a1432" containerID="8e1afa093cafdc000f2e4e1356fde48b992c93a6f30ad67564ad4d78aea903fd" exitCode=0 Oct 30 21:44:27.126283 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.126108 4335 generic.go:356] "Generic (PLEG): container finished" podID="1ac3165a-2ba0-4be9-8ab9-4d088d9a1432" containerID="77aaa48ef6d2906233a336132bd2c050cb5abbe4d48158ce23353b5f594b4255" exitCode=0 Oct 30 21:44:27.126283 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.126115 4335 generic.go:356] "Generic (PLEG): container finished" podID="1ac3165a-2ba0-4be9-8ab9-4d088d9a1432" containerID="d122bcb7e202204274baa3d7b8435ae85b2cd95774fb1f715c0ca864318a4b6a" exitCode=0 Oct 30 21:44:27.126283 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.126120 4335 generic.go:356] "Generic (PLEG): container finished" podID="1ac3165a-2ba0-4be9-8ab9-4d088d9a1432" containerID="c7abf33c058d51570f7345a9aa45cf0c0f230fef34da0d77274bd7e09f0cf74e" exitCode=0 Oct 30 21:44:27.129172 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.129152 4335 generic.go:356] "Generic (PLEG): container finished" podID="de7a233ef9a012d605524a0af5a13d0b" containerID="2d3b5a1cba68397b73022adb71163b8756aedbd710b0608af9fb0e64ca9547df" exitCode=0 Oct 30 21:44:27.131361 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.131343 4335 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:44:27.132288 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.132267 4335 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:44:27.132365 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.132303 4335 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:44:27.132365 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.132318 4335 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:44:27.133596 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.133581 4335 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:44:27.133596 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.133594 4335 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:44:27.133712 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.133608 4335 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:44:27.133755 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.133740 4335 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:44:27.133784 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.133757 4335 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:44:27.133784 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.133773 4335 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:44:27.133784 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.133778 4335 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:44:27.133784 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.133783 4335 policy_none.go:50] "Start" Oct 30 21:44:27.133887 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.133792 4335 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:44:27.133887 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.133802 4335 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:44:27.134002 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.133989 4335 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:44:27.134738 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.134722 4335 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:44:27.134815 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.134754 4335 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:44:27.134815 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.134769 4335 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:44:27.142104 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.142087 4335 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:44:27.142104 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.142106 4335 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:44:27.142193 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.142112 4335 policy_none.go:44] "Start" Oct 30 21:44:27.146461 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.146304 4335 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:44:27.150164 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:44:27.150145 4335 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 21:44:27.150890 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.150871 4335 manager.go:341] "Starting Device Plugin manager" Oct 30 21:44:27.150975 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:44:27.150894 4335 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:44:27.150975 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.150902 4335 server.go:87] "Starting device plugin registration server" Oct 30 21:44:27.151181 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.151165 4335 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:44:27.151238 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.151183 4335 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:44:27.151986 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.151390 4335 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:44:27.151986 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.151466 4335 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:44:27.151986 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.151495 4335 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:44:27.151986 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.151502 4335 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:44:27.151986 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.151581 4335 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:44:27.151486167Z","UUID":"4c5b9d97-7265-4c0c-82eb-a53bbbab6311","Handler":null,"Name":"","Endpoint":""} Oct 30 21:44:27.152959 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:44:27.152943 4335 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:44:27.153075 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:44:27.153064 4335 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-193.us-west-1.compute.internal\" not found" Oct 30 21:44:27.154268 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.154249 4335 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:44:27.154374 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.154274 4335 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:44:27.211290 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.211266 4335 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:44:27.211449 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.211432 4335 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:44:27.213380 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.213363 4335 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:44:27.213454 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.213365 4335 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:44:27.251440 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.251418 4335 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:44:27.252272 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.252256 4335 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:44:27.252344 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.252287 4335 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:44:27.252344 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.252303 4335 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:44:27.252344 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.252340 4335 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:27.263112 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.263082 4335 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:44:27.350835 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.350802 4335 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal"] Oct 30 21:44:27.351283 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.351244 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal" event={"ID":"8b76742bc66466f5dd9bc77bf5370e40","Type":"ContainerStarted","Data":"75d2e0ab2086dd6af6ee94f2f8926ce0296c9046b03d6a3faf2b724517a60fad"} Oct 30 21:44:27.351368 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.351296 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal" event={"ID":"8b76742bc66466f5dd9bc77bf5370e40","Type":"ContainerStarted","Data":"4d1b6a0958405e70a9f74d458070972d7112ad956c090e4ac078281e0474474a"} Oct 30 21:44:27.351368 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.351337 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" event={"ID":"de7a233ef9a012d605524a0af5a13d0b","Type":"ContainerStarted","Data":"4a74b113946f6d46cc889e5e67621f877d96d48fc5fc024e1421d3111959c2b0"} Oct 30 21:44:27.351368 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.351356 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" event={"ID":"de7a233ef9a012d605524a0af5a13d0b","Type":"ContainerDied","Data":"2d3b5a1cba68397b73022adb71163b8756aedbd710b0608af9fb0e64ca9547df"} Oct 30 21:44:27.351368 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.351364 4335 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:27.351553 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.351375 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" event={"ID":"de7a233ef9a012d605524a0af5a13d0b","Type":"ContainerStarted","Data":"03516dcf44f10b5c29eaccb99085929e2c26bbe9b1dd584f83a2034331c977fb"} Oct 30 21:44:27.351553 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.351457 4335 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:27.361809 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.361768 4335 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:27.361809 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.361809 4335 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:27.362194 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.362180 4335 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeNotReady" Oct 30 21:44:27.362244 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.362211 4335 setters.go:546] "Node became not ready" node="ip-10-0-131-193.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T21:44:27Z","lastTransitionTime":"2026-10-30T21:44:27Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 30 21:44:27.377660 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.377605 4335 nodeinfomanager.go:471] existing CSINode "ip-10-0-131-193.us-west-1.compute.internal" is owned by different node (oldNodeID="b55ea0eb-9a5e-4aa9-a392-4cc25890460e", newNodeID=""), cleaning up... Oct 30 21:44:27.411694 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.411674 4335 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:44:27.411772 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:44:27.411711 4335 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:27.412454 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.412437 4335 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:44:27.412490 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:44:27.412482 4335 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:27.415214 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.415198 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/de7a233ef9a012d605524a0af5a13d0b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal\" (UID: \"de7a233ef9a012d605524a0af5a13d0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:27.415254 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.415229 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de7a233ef9a012d605524a0af5a13d0b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal\" (UID: \"de7a233ef9a012d605524a0af5a13d0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:27.415254 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.415248 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b76742bc66466f5dd9bc77bf5370e40-config\") pod \"kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal\" (UID: \"8b76742bc66466f5dd9bc77bf5370e40\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:27.515920 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.515900 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/de7a233ef9a012d605524a0af5a13d0b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal\" (UID: \"de7a233ef9a012d605524a0af5a13d0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:27.516011 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.515924 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de7a233ef9a012d605524a0af5a13d0b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal\" (UID: \"de7a233ef9a012d605524a0af5a13d0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:27.516011 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.515940 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b76742bc66466f5dd9bc77bf5370e40-config\") pod \"kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal\" (UID: \"8b76742bc66466f5dd9bc77bf5370e40\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:27.516011 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.515977 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b76742bc66466f5dd9bc77bf5370e40-config\") pod \"kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal\" (UID: \"8b76742bc66466f5dd9bc77bf5370e40\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:27.516011 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.516005 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/de7a233ef9a012d605524a0af5a13d0b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal\" (UID: \"de7a233ef9a012d605524a0af5a13d0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:27.516145 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.516009 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de7a233ef9a012d605524a0af5a13d0b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal\" (UID: \"de7a233ef9a012d605524a0af5a13d0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:27.630988 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.630943 4335 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-193.us-west-1.compute.internal" event="NodeReady" Oct 30 21:44:27.631063 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.631006 4335 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:44:27.997799 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:27.997734 4335 apiserver.go:52] "Watching apiserver" Oct 30 21:44:28.133724 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.133691 4335 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:28.133841 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.133812 4335 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:28.149335 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.149314 4335 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:44:28.149821 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.149801 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-wh7rd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl","openshift-kube-storage-version-migrator/migrator-66df5d496b-nxf92","openshift-multus/multus-additional-cni-plugins-n6b4f","kube-system/konnectivity-agent-lpx2r","openshift-cluster-node-tuning-operator/tuned-89cpv","openshift-image-registry/node-ca-45jdt","openshift-insights/periodic-gathering-bcs5k-f5hhp","kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal","openshift-dns/node-resolver-vx4gh","openshift-insights/insights-runtime-extractor-f8pkp","kube-system/global-pull-secret-syncer-8mr69","openshift-multus/multus-b5fb6","openshift-ingress-canary/ingress-canary-tg5p8","openshift-network-operator/iptables-alerter-vkl9r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal","openshift-service-ca/service-ca-5fffc97bdf-2p9r5","openshift-dns/dns-default-sqfth","openshift-network-diagnostics/network-check-target-tcckj","openshift-ovn-kubernetes/ovnkube-node-w6p4f"] Oct 30 21:44:28.153950 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.153917 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:28.157721 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.157699 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:28.158060 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.158046 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:28.194013 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.193997 4335 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:44:28.194096 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:44:28.194037 4335 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:28.194631 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.194600 4335 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:44:28.194677 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:44:28.194652 4335 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.us-west-1.compute.internal" Oct 30 21:44:28.202599 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.202585 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:44:28.203163 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.203146 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:44:28.203286 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.203274 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:44:28.204425 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.204403 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:44:28.204425 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.204421 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:44:28.204640 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.204440 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:44:28.204640 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.204457 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:44:28.204640 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.204510 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:44:28.204640 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.204519 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:44:28.204640 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.204551 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:44:28.204640 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.204557 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:44:28.204640 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.204559 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:44:28.204640 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.204551 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:44:28.204640 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.204608 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:44:28.204887 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.204696 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:44:28.204887 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.204740 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:44:28.204887 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.204770 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:44:28.204887 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.204830 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:44:28.205078 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.204950 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:44:28.205955 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.205939 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:44:28.206116 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.206076 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:44:28.206380 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.206365 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:44:28.206498 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.206467 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:44:28.206831 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.206813 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:44:28.206932 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.206864 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vv25x\"" Oct 30 21:44:28.207141 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.207129 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:44:28.207141 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.207137 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hkrw6\"" Oct 30 21:44:28.207141 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.207138 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:44:28.207334 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.207320 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ph4bx\"" Oct 30 21:44:28.207409 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.207362 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:44:28.207409 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.207385 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9vv2h\"" Oct 30 21:44:28.207409 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.207401 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5pc99\"" Oct 30 21:44:28.207581 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.207414 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-22kpt\"" Oct 30 21:44:28.207581 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.207404 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fqpcl\"" Oct 30 21:44:28.207581 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.207415 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s7wrh\"" Oct 30 21:44:28.207581 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.207473 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:44:28.207581 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.207535 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:44:28.207814 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.207637 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j2rr9\"" Oct 30 21:44:28.207814 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.207638 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:44:28.207814 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.207689 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gmj65\"" Oct 30 21:44:28.207814 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.207710 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6ptf6\"" Oct 30 21:44:28.207814 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.207760 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6hfjp\"" Oct 30 21:44:28.211810 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.211794 4335 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:44:28.219102 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219082 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-var-lib-openvswitch\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.219176 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219114 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.219176 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219139 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-cnibin\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:44:28.219237 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219189 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-trusted-ca-bundle\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:28.219237 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219214 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-systemd-units\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.219327 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219276 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-node-log\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.219327 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219305 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-sysctl-conf\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.219384 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219342 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535-host-slash\") pod \"iptables-alerter-vkl9r\" (UID: \"4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535\") " pod="openshift-network-operator/iptables-alerter-vkl9r" Oct 30 21:44:28.219384 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219363 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/08df8d3b-f392-438a-81d5-3ee04ee2cd8b-agent-certs\") pod \"konnectivity-agent-lpx2r\" (UID: \"08df8d3b-f392-438a-81d5-3ee04ee2cd8b\") " pod="kube-system/konnectivity-agent-lpx2r" Oct 30 21:44:28.219384 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219379 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-registration-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.219488 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219426 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-modprobe-d\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.219519 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219513 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-tuned\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.219566 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219527 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12b346ba-5a09-4ec5-8c16-885fbe1502bc-cert\") pod \"ingress-canary-tg5p8\" (UID: \"12b346ba-5a09-4ec5-8c16-885fbe1502bc\") " pod="openshift-ingress-canary/ingress-canary-tg5p8" Oct 30 21:44:28.219602 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219570 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-run-k8s-cni-cncf-io\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.219682 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219665 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64p8r\" (UniqueName: \"kubernetes.io/projected/8f03953c-1f9f-4b43-8f1d-0a5f3766340b-kube-api-access-64p8r\") pod \"service-ca-5fffc97bdf-2p9r5\" (UID: \"8f03953c-1f9f-4b43-8f1d-0a5f3766340b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" Oct 30 21:44:28.219735 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219683 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-tuned\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.219735 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219699 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-run-systemd\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.219735 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219723 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/08df8d3b-f392-438a-81d5-3ee04ee2cd8b-agent-certs\") pod \"konnectivity-agent-lpx2r\" (UID: \"08df8d3b-f392-438a-81d5-3ee04ee2cd8b\") " pod="kube-system/konnectivity-agent-lpx2r" Oct 30 21:44:28.219823 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219796 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08685d97-2f18-4abe-a7b8-25466aee77cb-ovn-node-metrics-cert\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.219854 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219832 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f73329a3-7290-44a1-a72e-1fe72c0d0da5-kubelet-config\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:28.219900 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219885 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f73329a3-7290-44a1-a72e-1fe72c0d0da5-dbus\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:28.219932 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219922 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzbrt\" (UniqueName: \"kubernetes.io/projected/c94b6070-d303-4c76-8895-73cd37a2f985-kube-api-access-bzbrt\") pod \"network-metrics-daemon-wh7rd\" (UID: \"c94b6070-d303-4c76-8895-73cd37a2f985\") " pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:28.219966 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.219957 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-cni-netd\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.220036 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220021 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08685d97-2f18-4abe-a7b8-25466aee77cb-ovn-node-metrics-cert\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.220082 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220050 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-socket-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.220111 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220093 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cbbb\" (UniqueName: \"kubernetes.io/projected/e1668e0e-55e0-493c-a735-1ff0da66eda3-kube-api-access-5cbbb\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.220111 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220108 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-system-cni-dir\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.220172 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220124 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a899495-aa25-4538-8eef-5640503b466a-cni-binary-copy\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.220200 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220176 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-kubelet\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.220270 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220245 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-log-socket\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.220299 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220287 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08685d97-2f18-4abe-a7b8-25466aee77cb-ovnkube-script-lib\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.220334 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220322 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:44:28.220364 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220346 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:44:28.220396 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220364 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92da1846-1753-43a7-84a7-3f317e84675d-crio-socket\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:28.220482 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220410 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a899495-aa25-4538-8eef-5640503b466a-cni-binary-copy\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.220482 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220413 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-cnibin\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.220552 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220482 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a899495-aa25-4538-8eef-5640503b466a-multus-daemon-config\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.220552 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220493 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08685d97-2f18-4abe-a7b8-25466aee77cb-ovnkube-script-lib\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.220552 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220508 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gv95\" (UniqueName: \"kubernetes.io/projected/000931e4-5bd3-4387-adf7-ea47151527ad-kube-api-access-9gv95\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:28.220683 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220549 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbn2h\" (UniqueName: \"kubernetes.io/projected/08685d97-2f18-4abe-a7b8-25466aee77cb-kube-api-access-hbn2h\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.220683 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220574 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee354605-03c9-4c69-a937-5f94440dc0ae-host\") pod \"node-ca-45jdt\" (UID: \"ee354605-03c9-4c69-a937-5f94440dc0ae\") " pod="openshift-image-registry/node-ca-45jdt" Oct 30 21:44:28.220683 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220578 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:44:28.220683 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220588 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:44:28.220683 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220598 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cgbv\" (UniqueName: \"kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv\") pod \"network-check-target-tcckj\" (UID: \"3ee9e6d0-f122-4ff3-aa72-b597339928ac\") " pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:44:28.220683 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220650 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:28.220948 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220685 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs\") pod \"network-metrics-daemon-wh7rd\" (UID: \"c94b6070-d303-4c76-8895-73cd37a2f985\") " pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:28.220948 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220721 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-var-lib-cni-multus\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.220948 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220750 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a899495-aa25-4538-8eef-5640503b466a-multus-daemon-config\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.220948 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220761 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-var-lib-kubelet\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.220948 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220804 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-etc-openvswitch\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.220948 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220846 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-cni-bin\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.220948 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220871 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:44:28.220948 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220887 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f73329a3-7290-44a1-a72e-1fe72c0d0da5-original-pull-secret\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:28.220948 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220926 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wfjl\" (UniqueName: \"kubernetes.io/projected/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-kube-api-access-4wfjl\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:44:28.221320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220953 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/08df8d3b-f392-438a-81d5-3ee04ee2cd8b-konnectivity-ca\") pod \"konnectivity-agent-lpx2r\" (UID: \"08df8d3b-f392-438a-81d5-3ee04ee2cd8b\") " pod="kube-system/konnectivity-agent-lpx2r" Oct 30 21:44:28.221320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.220994 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d45916e8-c498-4bf5-b5f3-df697b608ed9-archives-path\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:28.221320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221018 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/827f9fb3-0eb3-4f8f-96f2-72df80c024d8-hosts-file\") pod \"node-resolver-vx4gh\" (UID: \"827f9fb3-0eb3-4f8f-96f2-72df80c024d8\") " pod="openshift-dns/node-resolver-vx4gh" Oct 30 21:44:28.221320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221039 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-multus-cni-dir\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.221320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221054 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d45916e8-c498-4bf5-b5f3-df697b608ed9-archives-path\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:28.221320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221063 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-var-lib-cni-bin\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.221320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221100 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-run-multus-certs\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.221320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221129 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-service-ca-bundle\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:28.221320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221152 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-device-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.221320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221176 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-etc-selinux\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.221320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221186 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/08df8d3b-f392-438a-81d5-3ee04ee2cd8b-konnectivity-ca\") pod \"konnectivity-agent-lpx2r\" (UID: \"08df8d3b-f392-438a-81d5-3ee04ee2cd8b\") " pod="kube-system/konnectivity-agent-lpx2r" Oct 30 21:44:28.221320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221218 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq6b2\" (UniqueName: \"kubernetes.io/projected/12b346ba-5a09-4ec5-8c16-885fbe1502bc-kube-api-access-pq6b2\") pod \"ingress-canary-tg5p8\" (UID: \"12b346ba-5a09-4ec5-8c16-885fbe1502bc\") " pod="openshift-ingress-canary/ingress-canary-tg5p8" Oct 30 21:44:28.221320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221237 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-multus-socket-dir-parent\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.221320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221299 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-run-netns\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.221867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221341 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:44:28.221867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221375 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92da1846-1753-43a7-84a7-3f317e84675d-data-volume\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:28.221867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221474 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-sys-fs\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.221867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221491 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-sysctl-d\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.221867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221558 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-lib-modules\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.221867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221570 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92da1846-1753-43a7-84a7-3f317e84675d-data-volume\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:28.221867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221596 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827f9fb3-0eb3-4f8f-96f2-72df80c024d8-tmp-dir\") pod \"node-resolver-vx4gh\" (UID: \"827f9fb3-0eb3-4f8f-96f2-72df80c024d8\") " pod="openshift-dns/node-resolver-vx4gh" Oct 30 21:44:28.221867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221650 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvscc\" (UniqueName: \"kubernetes.io/projected/3a846703-92eb-4115-875f-c347fc4f2a1c-kube-api-access-tvscc\") pod \"migrator-66df5d496b-nxf92\" (UID: \"3a846703-92eb-4115-875f-c347fc4f2a1c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxf92" Oct 30 21:44:28.221867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221673 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827f9fb3-0eb3-4f8f-96f2-72df80c024d8-tmp-dir\") pod \"node-resolver-vx4gh\" (UID: \"827f9fb3-0eb3-4f8f-96f2-72df80c024d8\") " pod="openshift-dns/node-resolver-vx4gh" Oct 30 21:44:28.221867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221676 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8f03953c-1f9f-4b43-8f1d-0a5f3766340b-signing-cabundle\") pod \"service-ca-5fffc97bdf-2p9r5\" (UID: \"8f03953c-1f9f-4b43-8f1d-0a5f3766340b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" Oct 30 21:44:28.221867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221702 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-run-ovn\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.221867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221725 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.221867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221759 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-kubernetes\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.221867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221802 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-system-cni-dir\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:44:28.221867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221831 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92da1846-1753-43a7-84a7-3f317e84675d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:28.221867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221849 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-sysconfig\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.222330 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221921 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-systemd\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.222330 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.221977 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/000931e4-5bd3-4387-adf7-ea47151527ad-config-volume\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:28.222330 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222006 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535-iptables-alerter-script\") pod \"iptables-alerter-vkl9r\" (UID: \"4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535\") " pod="openshift-network-operator/iptables-alerter-vkl9r" Oct 30 21:44:28.222330 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222032 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:44:28.222330 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222032 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:28.222330 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222074 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-run-netns\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.222330 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222094 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08685d97-2f18-4abe-a7b8-25466aee77cb-env-overrides\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.222330 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222142 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrgmx\" (UniqueName: \"kubernetes.io/projected/d45916e8-c498-4bf5-b5f3-df697b608ed9-kube-api-access-lrgmx\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:28.222330 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222178 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-var-lib-kubelet\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.222330 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222255 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08685d97-2f18-4abe-a7b8-25466aee77cb-env-overrides\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.222330 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222269 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46rpn\" (UniqueName: \"kubernetes.io/projected/74fe5a17-9648-48a9-8d1b-338990dac55b-kube-api-access-46rpn\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.222330 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222300 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-hostroot\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.222787 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222360 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8f03953c-1f9f-4b43-8f1d-0a5f3766340b-signing-key\") pod \"service-ca-5fffc97bdf-2p9r5\" (UID: \"8f03953c-1f9f-4b43-8f1d-0a5f3766340b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" Oct 30 21:44:28.222787 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222388 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nt72\" (UniqueName: \"kubernetes.io/projected/ee354605-03c9-4c69-a937-5f94440dc0ae-kube-api-access-7nt72\") pod \"node-ca-45jdt\" (UID: \"ee354605-03c9-4c69-a937-5f94440dc0ae\") " pod="openshift-image-registry/node-ca-45jdt" Oct 30 21:44:28.222787 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222420 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.222787 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222528 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-run\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.222787 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222556 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-host\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.222787 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222636 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9qk6\" (UniqueName: \"kubernetes.io/projected/827f9fb3-0eb3-4f8f-96f2-72df80c024d8-kube-api-access-v9qk6\") pod \"node-resolver-vx4gh\" (UID: \"827f9fb3-0eb3-4f8f-96f2-72df80c024d8\") " pod="openshift-dns/node-resolver-vx4gh" Oct 30 21:44:28.222787 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222680 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-slash\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.222787 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222723 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-run-openvswitch\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.222787 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222766 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee354605-03c9-4c69-a937-5f94440dc0ae-serviceca\") pod \"node-ca-45jdt\" (UID: \"ee354605-03c9-4c69-a937-5f94440dc0ae\") " pod="openshift-image-registry/node-ca-45jdt" Oct 30 21:44:28.223185 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222801 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:28.223185 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222862 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7k9p\" (UniqueName: \"kubernetes.io/projected/92da1846-1753-43a7-84a7-3f317e84675d-kube-api-access-m7k9p\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:28.223185 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222895 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-sys\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.223185 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222919 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjtfp\" (UniqueName: \"kubernetes.io/projected/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535-kube-api-access-hjtfp\") pod \"iptables-alerter-vkl9r\" (UID: \"4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535\") " pod="openshift-network-operator/iptables-alerter-vkl9r" Oct 30 21:44:28.223185 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222948 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/000931e4-5bd3-4387-adf7-ea47151527ad-tmp-dir\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:28.223185 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222966 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08685d97-2f18-4abe-a7b8-25466aee77cb-ovnkube-config\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.223185 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.222982 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74fe5a17-9648-48a9-8d1b-338990dac55b-tmp\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.223185 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.223001 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-multus-conf-dir\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.223185 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.223018 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-etc-kubernetes\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.223185 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.223035 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8l9c\" (UniqueName: \"kubernetes.io/projected/2a899495-aa25-4538-8eef-5640503b466a-kube-api-access-p8l9c\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.223185 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.223000 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee354605-03c9-4c69-a937-5f94440dc0ae-serviceca\") pod \"node-ca-45jdt\" (UID: \"ee354605-03c9-4c69-a937-5f94440dc0ae\") " pod="openshift-image-registry/node-ca-45jdt" Oct 30 21:44:28.223514 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.223204 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74fe5a17-9648-48a9-8d1b-338990dac55b-tmp\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.223514 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.223239 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08685d97-2f18-4abe-a7b8-25466aee77cb-ovnkube-config\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.223514 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.223254 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/000931e4-5bd3-4387-adf7-ea47151527ad-tmp-dir\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:28.243309 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.243288 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:44:28.251627 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.251519 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:44:28.251627 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.251538 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:44:28.251750 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.251703 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:44:28.251750 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.251747 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:44:28.251907 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.251839 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vzpnp\"" Oct 30 21:44:28.251907 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.251887 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qjn2k\"" Oct 30 21:44:28.252019 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.251991 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:44:28.252059 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.252051 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:44:28.252266 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.252252 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:44:28.252266 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.252257 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535-iptables-alerter-script\") pod \"iptables-alerter-vkl9r\" (UID: \"4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535\") " pod="openshift-network-operator/iptables-alerter-vkl9r" Oct 30 21:44:28.252347 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.252330 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/000931e4-5bd3-4387-adf7-ea47151527ad-config-volume\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:28.252703 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.252686 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-4bwnp\"" Oct 30 21:44:28.252778 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.252730 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ltjsw\"" Oct 30 21:44:28.253226 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.253213 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-f7x6x\"" Oct 30 21:44:28.253686 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.253670 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 21:44:28.253749 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.253681 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:44:28.253749 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.253717 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:44:28.253888 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.253833 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sqd9j\"" Oct 30 21:44:28.253888 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.253862 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 21:44:28.253888 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.253882 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:44:28.254309 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.253917 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 21:44:28.254385 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.254328 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:44:28.254437 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.254394 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 21:44:28.256454 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.256439 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:44:28.256790 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.256777 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:44:28.262884 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.260300 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12b346ba-5a09-4ec5-8c16-885fbe1502bc-cert\") pod \"ingress-canary-tg5p8\" (UID: \"12b346ba-5a09-4ec5-8c16-885fbe1502bc\") " pod="openshift-ingress-canary/ingress-canary-tg5p8" Oct 30 21:44:28.262884 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.260606 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-trusted-ca-bundle\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:28.263042 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.263008 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8f03953c-1f9f-4b43-8f1d-0a5f3766340b-signing-cabundle\") pod \"service-ca-5fffc97bdf-2p9r5\" (UID: \"8f03953c-1f9f-4b43-8f1d-0a5f3766340b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" Oct 30 21:44:28.263132 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.263058 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8f03953c-1f9f-4b43-8f1d-0a5f3766340b-signing-key\") pod \"service-ca-5fffc97bdf-2p9r5\" (UID: \"8f03953c-1f9f-4b43-8f1d-0a5f3766340b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" Oct 30 21:44:28.263460 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.263441 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92da1846-1753-43a7-84a7-3f317e84675d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:28.263666 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.263648 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-service-ca-bundle\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:28.277959 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.277938 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cbbb\" (UniqueName: \"kubernetes.io/projected/e1668e0e-55e0-493c-a735-1ff0da66eda3-kube-api-access-5cbbb\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.278210 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.278190 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzbrt\" (UniqueName: \"kubernetes.io/projected/c94b6070-d303-4c76-8895-73cd37a2f985-kube-api-access-bzbrt\") pod \"network-metrics-daemon-wh7rd\" (UID: \"c94b6070-d303-4c76-8895-73cd37a2f985\") " pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:28.278316 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.278302 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64p8r\" (UniqueName: \"kubernetes.io/projected/8f03953c-1f9f-4b43-8f1d-0a5f3766340b-kube-api-access-64p8r\") pod \"service-ca-5fffc97bdf-2p9r5\" (UID: \"8f03953c-1f9f-4b43-8f1d-0a5f3766340b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2p9r5" Oct 30 21:44:28.278837 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.278817 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gv95\" (UniqueName: \"kubernetes.io/projected/000931e4-5bd3-4387-adf7-ea47151527ad-kube-api-access-9gv95\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:28.279037 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.279022 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq6b2\" (UniqueName: \"kubernetes.io/projected/12b346ba-5a09-4ec5-8c16-885fbe1502bc-kube-api-access-pq6b2\") pod \"ingress-canary-tg5p8\" (UID: \"12b346ba-5a09-4ec5-8c16-885fbe1502bc\") " pod="openshift-ingress-canary/ingress-canary-tg5p8" Oct 30 21:44:28.279126 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.279112 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cgbv\" (UniqueName: \"kubernetes.io/projected/3ee9e6d0-f122-4ff3-aa72-b597339928ac-kube-api-access-5cgbv\") pod \"network-check-target-tcckj\" (UID: \"3ee9e6d0-f122-4ff3-aa72-b597339928ac\") " pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:44:28.279156 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.279121 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbn2h\" (UniqueName: \"kubernetes.io/projected/08685d97-2f18-4abe-a7b8-25466aee77cb-kube-api-access-hbn2h\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.279186 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.279123 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wfjl\" (UniqueName: \"kubernetes.io/projected/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-kube-api-access-4wfjl\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:44:28.280124 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.280099 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvscc\" (UniqueName: \"kubernetes.io/projected/3a846703-92eb-4115-875f-c347fc4f2a1c-kube-api-access-tvscc\") pod \"migrator-66df5d496b-nxf92\" (UID: \"3a846703-92eb-4115-875f-c347fc4f2a1c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxf92" Oct 30 21:44:28.280915 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.280897 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nt72\" (UniqueName: \"kubernetes.io/projected/ee354605-03c9-4c69-a937-5f94440dc0ae-kube-api-access-7nt72\") pod \"node-ca-45jdt\" (UID: \"ee354605-03c9-4c69-a937-5f94440dc0ae\") " pod="openshift-image-registry/node-ca-45jdt" Oct 30 21:44:28.280992 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.280897 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46rpn\" (UniqueName: \"kubernetes.io/projected/74fe5a17-9648-48a9-8d1b-338990dac55b-kube-api-access-46rpn\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.283033 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.283015 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjtfp\" (UniqueName: \"kubernetes.io/projected/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535-kube-api-access-hjtfp\") pod \"iptables-alerter-vkl9r\" (UID: \"4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535\") " pod="openshift-network-operator/iptables-alerter-vkl9r" Oct 30 21:44:28.283228 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.283212 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8l9c\" (UniqueName: \"kubernetes.io/projected/2a899495-aa25-4538-8eef-5640503b466a-kube-api-access-p8l9c\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.283432 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.283417 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7k9p\" (UniqueName: \"kubernetes.io/projected/92da1846-1753-43a7-84a7-3f317e84675d-kube-api-access-m7k9p\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:28.284123 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.284108 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9qk6\" (UniqueName: \"kubernetes.io/projected/827f9fb3-0eb3-4f8f-96f2-72df80c024d8-kube-api-access-v9qk6\") pod \"node-resolver-vx4gh\" (UID: \"827f9fb3-0eb3-4f8f-96f2-72df80c024d8\") " pod="openshift-dns/node-resolver-vx4gh" Oct 30 21:44:28.290321 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.290305 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrgmx\" (UniqueName: \"kubernetes.io/projected/d45916e8-c498-4bf5-b5f3-df697b608ed9-kube-api-access-lrgmx\") pod \"periodic-gathering-bcs5k-f5hhp\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:44:28.324308 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324283 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:28.324390 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324311 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-run-netns\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.324390 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324328 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-var-lib-kubelet\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.324390 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324342 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-hostroot\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.324390 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324358 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.324556 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324390 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-run\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.324556 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324404 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.324556 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324395 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-run-netns\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.324556 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324424 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-host\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.324556 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324463 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-var-lib-kubelet\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.324556 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324415 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-hostroot\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.324556 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324487 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-run\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.324556 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324471 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-host\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.324556 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324500 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-slash\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.324556 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324465 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-slash\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.324556 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324525 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-run-openvswitch\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.324556 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324542 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324581 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-run-openvswitch\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324650 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-sys\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324680 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-multus-conf-dir\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324714 4335 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324720 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-multus-conf-dir\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324730 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-etc-kubernetes\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324761 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-sys\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324770 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-var-lib-openvswitch\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324793 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-etc-kubernetes\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324796 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324834 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324837 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-cnibin\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324857 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-var-lib-openvswitch\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324879 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-cnibin\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324885 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-systemd-units\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324921 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-node-log\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324928 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-systemd-units\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324937 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-sysctl-conf\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324967 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-node-log\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.324976 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535-host-slash\") pod \"iptables-alerter-vkl9r\" (UID: \"4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535\") " pod="openshift-network-operator/iptables-alerter-vkl9r" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325000 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-registration-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325007 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535-host-slash\") pod \"iptables-alerter-vkl9r\" (UID: \"4a34a83c-e6d4-42c4-8bd6-0c39fbcdf535\") " pod="openshift-network-operator/iptables-alerter-vkl9r" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325019 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-modprobe-d\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325040 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-run-k8s-cni-cncf-io\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325060 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-sysctl-conf\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.325049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325055 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-registration-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325063 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-run-systemd\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325095 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f73329a3-7290-44a1-a72e-1fe72c0d0da5-kubelet-config\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325109 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-run-systemd\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325117 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f73329a3-7290-44a1-a72e-1fe72c0d0da5-dbus\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325159 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-run-k8s-cni-cncf-io\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325166 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-cni-netd\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325175 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f73329a3-7290-44a1-a72e-1fe72c0d0da5-kubelet-config\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325190 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-socket-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325136 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-modprobe-d\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325215 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-cni-netd\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325206 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-system-cni-dir\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325259 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f73329a3-7290-44a1-a72e-1fe72c0d0da5-dbus\") pod \"global-pull-secret-syncer-8mr69\" (UID: \"f73329a3-7290-44a1-a72e-1fe72c0d0da5\") " pod="kube-system/global-pull-secret-syncer-8mr69" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325291 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-socket-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325337 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-system-cni-dir\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325341 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-kubelet\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325374 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-kubelet\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325384 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-log-socket\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325401 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92da1846-1753-43a7-84a7-3f317e84675d-crio-socket\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325425 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-cnibin\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325447 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee354605-03c9-4c69-a937-5f94440dc0ae-host\") pod \"node-ca-45jdt\" (UID: \"ee354605-03c9-4c69-a937-5f94440dc0ae\") " pod="openshift-image-registry/node-ca-45jdt" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325456 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92da1846-1753-43a7-84a7-3f317e84675d-crio-socket\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325461 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-log-socket\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325471 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs\") pod \"network-metrics-daemon-wh7rd\" (UID: \"c94b6070-d303-4c76-8895-73cd37a2f985\") " pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325491 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-var-lib-cni-multus\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325506 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-var-lib-kubelet\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325507 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee354605-03c9-4c69-a937-5f94440dc0ae-host\") pod \"node-ca-45jdt\" (UID: \"ee354605-03c9-4c69-a937-5f94440dc0ae\") " pod="openshift-image-registry/node-ca-45jdt" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325521 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-etc-openvswitch\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325526 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-var-lib-cni-multus\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325492 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-cnibin\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325544 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-cni-bin\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:44:28.325548 4335 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325552 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-var-lib-kubelet\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325559 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-etc-openvswitch\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325564 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325582 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/827f9fb3-0eb3-4f8f-96f2-72df80c024d8-hosts-file\") pod \"node-resolver-vx4gh\" (UID: \"827f9fb3-0eb3-4f8f-96f2-72df80c024d8\") " pod="openshift-dns/node-resolver-vx4gh" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325591 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-cni-bin\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325596 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-multus-cni-dir\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:44:28.325632 4335 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs podName:c94b6070-d303-4c76-8895-73cd37a2f985 nodeName:}" failed. No retries permitted until 2026-10-30 21:44:28.825595105 +0000 UTC m=+1.926026852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs") pod "network-metrics-daemon-wh7rd" (UID: "c94b6070-d303-4c76-8895-73cd37a2f985") : secret "metrics-daemon-secret" not found Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325659 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325665 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-multus-cni-dir\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325663 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-var-lib-cni-bin\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325673 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/827f9fb3-0eb3-4f8f-96f2-72df80c024d8-hosts-file\") pod \"node-resolver-vx4gh\" (UID: \"827f9fb3-0eb3-4f8f-96f2-72df80c024d8\") " pod="openshift-dns/node-resolver-vx4gh" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325694 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-run-multus-certs\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325698 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-var-lib-cni-bin\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325715 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-device-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325733 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-run-multus-certs\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325738 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-etc-selinux\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325782 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-multus-socket-dir-parent\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325807 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-etc-selinux\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325808 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-run-netns\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325831 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-sys-fs\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325837 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-host-run-netns\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325784 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-device-dir\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325849 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-sysctl-d\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325867 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-lib-modules\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325907 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1668e0e-55e0-493c-a735-1ff0da66eda3-sys-fs\") pod \"aws-ebs-csi-driver-node-2plkl\" (UID: \"e1668e0e-55e0-493c-a735-1ff0da66eda3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2plkl" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325928 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-run-ovn\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325942 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-lib-modules\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325945 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a899495-aa25-4538-8eef-5640503b466a-multus-socket-dir-parent\") pod \"multus-b5fb6\" (UID: \"2a899495-aa25-4538-8eef-5640503b466a\") " pod="openshift-multus/multus-b5fb6" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325953 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325975 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-run-ovn\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325995 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-kubernetes\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.326025 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-system-cni-dir\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.326036 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-kubernetes\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.326000 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-sysctl-d\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.326058 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-sysconfig\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.326083 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ac3165a-2ba0-4be9-8ab9-4d088d9a1432-system-cni-dir\") pod \"multus-additional-cni-plugins-n6b4f\" (UID: \"1ac3165a-2ba0-4be9-8ab9-4d088d9a1432\") " pod="openshift-multus/multus-additional-cni-plugins-n6b4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.325982 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08685d97-2f18-4abe-a7b8-25466aee77cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6p4f\" (UID: \"08685d97-2f18-4abe-a7b8-25466aee77cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.326098 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-sysconfig\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.326090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.326100 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-systemd\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.328199 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.326165 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74fe5a17-9648-48a9-8d1b-338990dac55b-etc-systemd\") pod \"tuned-89cpv\" (UID: \"74fe5a17-9648-48a9-8d1b-338990dac55b\") " pod="openshift-cluster-node-tuning-operator/tuned-89cpv" Oct 30 21:44:28.328199 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.327997 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/000931e4-5bd3-4387-adf7-ea47151527ad-metrics-tls\") pod \"dns-default-sqfth\" (UID: \"000931e4-5bd3-4387-adf7-ea47151527ad\") " pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:28.328199 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.328012 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92da1846-1753-43a7-84a7-3f317e84675d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8pkp\" (UID: \"92da1846-1753-43a7-84a7-3f317e84675d\") " pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:28.393659 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.393637 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tg5p8_12b346ba-5a09-4ec5-8c16-885fbe1502bc/serve-healthcheck-canary/0.log" Oct 30 21:44:28.458277 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.458260 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f8pkp" Oct 30 21:44:28.458277 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.458272 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:28.832855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.832700 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs\") pod \"network-metrics-daemon-wh7rd\" (UID: \"c94b6070-d303-4c76-8895-73cd37a2f985\") " pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:28.832855 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:44:28.832846 4335 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:44:28.833081 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:44:28.832936 4335 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs podName:c94b6070-d303-4c76-8895-73cd37a2f985 nodeName:}" failed. No retries permitted until 2026-10-30 21:44:29.832915945 +0000 UTC m=+2.933347669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs") pod "network-metrics-daemon-wh7rd" (UID: "c94b6070-d303-4c76-8895-73cd37a2f985") : secret "metrics-daemon-secret" not found Oct 30 21:44:28.901117 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.901083 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f8pkp"] Oct 30 21:44:28.905237 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:28.905208 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92da1846_1753_43a7_84a7_3f317e84675d.slice/crio-a12ff6b0eb0fc72cd76f6e358dbe489f1a04922caeaad3179154208251963179 WatchSource:0}: Error finding container a12ff6b0eb0fc72cd76f6e358dbe489f1a04922caeaad3179154208251963179: Status 404 returned error can't find the container with id a12ff6b0eb0fc72cd76f6e358dbe489f1a04922caeaad3179154208251963179 Oct 30 21:44:28.931519 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.931501 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sqfth"] Oct 30 21:44:28.935676 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:28.935655 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000931e4_5bd3_4387_adf7_ea47151527ad.slice/crio-c339b7d14780959520ee070bbceb21292285b691eed94a31ce2486a09e481ebf WatchSource:0}: Error finding container c339b7d14780959520ee070bbceb21292285b691eed94a31ce2486a09e481ebf: Status 404 returned error can't find the container with id c339b7d14780959520ee070bbceb21292285b691eed94a31ce2486a09e481ebf Oct 30 21:44:28.937591 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.937577 4335 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:44:28.993627 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:28.993598 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-nxf92_3a846703-92eb-4115-875f-c347fc4f2a1c/migrator/0.log" Oct 30 21:44:29.137876 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:29.137815 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sqfth" event={"ID":"000931e4-5bd3-4387-adf7-ea47151527ad","Type":"ContainerStarted","Data":"c339b7d14780959520ee070bbceb21292285b691eed94a31ce2486a09e481ebf"} Oct 30 21:44:29.139485 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:29.139462 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8pkp" event={"ID":"92da1846-1753-43a7-84a7-3f317e84675d","Type":"ContainerStarted","Data":"7e5cf734165906d94b49a4666318bb875eb1728380466040f1d4a03eb8a5e092"} Oct 30 21:44:29.139750 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:29.139729 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8pkp" event={"ID":"92da1846-1753-43a7-84a7-3f317e84675d","Type":"ContainerStarted","Data":"a12ff6b0eb0fc72cd76f6e358dbe489f1a04922caeaad3179154208251963179"} Oct 30 21:44:29.840113 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:29.840087 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs\") pod \"network-metrics-daemon-wh7rd\" (UID: \"c94b6070-d303-4c76-8895-73cd37a2f985\") " pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:29.843528 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:29.843507 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94b6070-d303-4c76-8895-73cd37a2f985-metrics-certs\") pod \"network-metrics-daemon-wh7rd\" (UID: \"c94b6070-d303-4c76-8895-73cd37a2f985\") " pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:29.954638 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:29.954603 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wh7rd" Oct 30 21:44:30.005190 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:30.005160 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:44:30.337912 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:30.337868 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wh7rd"] Oct 30 21:44:30.341952 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:30.341916 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94b6070_d303_4c76_8895_73cd37a2f985.slice/crio-def0a74aaf7166d920ccc43d5eee9e36b5321deed44e4e254d6a3c4c62cd88d2 WatchSource:0}: Error finding container def0a74aaf7166d920ccc43d5eee9e36b5321deed44e4e254d6a3c4c62cd88d2: Status 404 returned error can't find the container with id def0a74aaf7166d920ccc43d5eee9e36b5321deed44e4e254d6a3c4c62cd88d2 Oct 30 21:44:31.147454 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:31.147421 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wh7rd" event={"ID":"c94b6070-d303-4c76-8895-73cd37a2f985","Type":"ContainerStarted","Data":"def0a74aaf7166d920ccc43d5eee9e36b5321deed44e4e254d6a3c4c62cd88d2"} Oct 30 21:44:31.149156 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:31.149129 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8pkp" event={"ID":"92da1846-1753-43a7-84a7-3f317e84675d","Type":"ContainerStarted","Data":"6a4c6e403a8d696199459a6ce6c3270b4dc5b5adfa8d0f2b4bf260687b01fdf5"} Oct 30 21:44:32.156144 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:32.155637 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sqfth" event={"ID":"000931e4-5bd3-4387-adf7-ea47151527ad","Type":"ContainerStarted","Data":"501b26311d155116301e38b99eedc1bb9f1b786989a5147261f986c268ed907a"} Oct 30 21:44:32.156144 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:32.155928 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:32.156144 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:32.155959 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sqfth" event={"ID":"000931e4-5bd3-4387-adf7-ea47151527ad","Type":"ContainerStarted","Data":"5bd79a49c0a4fdf7b92cf373719307f9444eefd9d5c2da2c02fe7e060de94060"} Oct 30 21:44:33.163770 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:33.163741 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8pkp" event={"ID":"92da1846-1753-43a7-84a7-3f317e84675d","Type":"ContainerStarted","Data":"c90c1083c2cb81aa8b61e19ab15262b6d7d31bc029ddb4f001f626a663f01977"} Oct 30 21:44:33.165542 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:33.165521 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wh7rd" event={"ID":"c94b6070-d303-4c76-8895-73cd37a2f985","Type":"ContainerStarted","Data":"da811e42f1d24ec583d1de923b4d730080e3aba32be872253aad644d7ad45d3b"} Oct 30 21:44:33.165688 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:33.165561 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wh7rd" event={"ID":"c94b6070-d303-4c76-8895-73cd37a2f985","Type":"ContainerStarted","Data":"2f13c5323720d66cf47974681e7435ac2ed5c6da1de2275611a6ab1ec39f889d"} Oct 30 21:44:34.539474 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:34.539292 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:34.804473 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:34.804445 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:35.172396 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:35.172340 4335 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:44:39.122903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:39.122872 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6p4f" Oct 30 21:44:39.212953 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:39.212923 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lpx2r" Oct 30 21:44:41.313036 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.313005 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-rkbjf"] Oct 30 21:44:41.314958 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.314940 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-rkbjf" Oct 30 21:44:41.378954 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.378928 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:44:41.407768 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.407747 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:44:41.407886 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.407820 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-zw8wl\"" Oct 30 21:44:41.415602 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.415574 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9274080-66f5-4497-8e0d-1bb850db463e-tmp\") pod \"downloads-6dd789c8b7-rkbjf\" (UID: \"b9274080-66f5-4497-8e0d-1bb850db463e\") " pod="openshift-console/downloads-6dd789c8b7-rkbjf" Oct 30 21:44:41.415703 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.415602 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksvx5\" (UniqueName: \"kubernetes.io/projected/b9274080-66f5-4497-8e0d-1bb850db463e-kube-api-access-ksvx5\") pod \"downloads-6dd789c8b7-rkbjf\" (UID: \"b9274080-66f5-4497-8e0d-1bb850db463e\") " pod="openshift-console/downloads-6dd789c8b7-rkbjf" Oct 30 21:44:41.418832 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.418816 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56fd84f888-knnp4"] Oct 30 21:44:41.420442 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.420429 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.477636 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.477590 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56fd84f888-knnp4"] Oct 30 21:44:41.516585 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.516562 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9274080-66f5-4497-8e0d-1bb850db463e-tmp\") pod \"downloads-6dd789c8b7-rkbjf\" (UID: \"b9274080-66f5-4497-8e0d-1bb850db463e\") " pod="openshift-console/downloads-6dd789c8b7-rkbjf" Oct 30 21:44:41.516784 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.516602 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksvx5\" (UniqueName: \"kubernetes.io/projected/b9274080-66f5-4497-8e0d-1bb850db463e-kube-api-access-ksvx5\") pod \"downloads-6dd789c8b7-rkbjf\" (UID: \"b9274080-66f5-4497-8e0d-1bb850db463e\") " pod="openshift-console/downloads-6dd789c8b7-rkbjf" Oct 30 21:44:41.516784 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.516673 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11d42b1f-609e-4901-9b31-76a2426fc294-image-registry-private-configuration\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.516784 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.516700 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11d42b1f-609e-4901-9b31-76a2426fc294-ca-trust-extracted\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.516784 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.516723 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11d42b1f-609e-4901-9b31-76a2426fc294-registry-tls\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.516784 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.516751 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11d42b1f-609e-4901-9b31-76a2426fc294-registry-certificates\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.517109 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.516853 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11d42b1f-609e-4901-9b31-76a2426fc294-bound-sa-token\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.517109 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.516954 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11d42b1f-609e-4901-9b31-76a2426fc294-installation-pull-secrets\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.517109 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.517004 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11d42b1f-609e-4901-9b31-76a2426fc294-trusted-ca\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.517109 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.517046 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvmkq\" (UniqueName: \"kubernetes.io/projected/11d42b1f-609e-4901-9b31-76a2426fc294-kube-api-access-vvmkq\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.517109 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.517096 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9274080-66f5-4497-8e0d-1bb850db463e-tmp\") pod \"downloads-6dd789c8b7-rkbjf\" (UID: \"b9274080-66f5-4497-8e0d-1bb850db463e\") " pod="openshift-console/downloads-6dd789c8b7-rkbjf" Oct 30 21:44:41.533988 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.533958 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:44:41.533988 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.533917 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:44:41.534146 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.534057 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:44:41.534146 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.534106 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h7hc5\"" Oct 30 21:44:41.534246 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.534160 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:44:41.534790 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.534773 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-rkbjf"] Oct 30 21:44:41.579910 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.579863 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksvx5\" (UniqueName: \"kubernetes.io/projected/b9274080-66f5-4497-8e0d-1bb850db463e-kube-api-access-ksvx5\") pod \"downloads-6dd789c8b7-rkbjf\" (UID: \"b9274080-66f5-4497-8e0d-1bb850db463e\") " pod="openshift-console/downloads-6dd789c8b7-rkbjf" Oct 30 21:44:41.617944 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.617923 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11d42b1f-609e-4901-9b31-76a2426fc294-registry-tls\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.618025 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.617951 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11d42b1f-609e-4901-9b31-76a2426fc294-registry-certificates\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.618025 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.617968 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11d42b1f-609e-4901-9b31-76a2426fc294-bound-sa-token\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.618025 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.617985 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11d42b1f-609e-4901-9b31-76a2426fc294-installation-pull-secrets\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.618025 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.618006 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11d42b1f-609e-4901-9b31-76a2426fc294-trusted-ca\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.618025 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.618026 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvmkq\" (UniqueName: \"kubernetes.io/projected/11d42b1f-609e-4901-9b31-76a2426fc294-kube-api-access-vvmkq\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.618278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.618088 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11d42b1f-609e-4901-9b31-76a2426fc294-image-registry-private-configuration\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.618278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.618133 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11d42b1f-609e-4901-9b31-76a2426fc294-ca-trust-extracted\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.618490 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.618472 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11d42b1f-609e-4901-9b31-76a2426fc294-ca-trust-extracted\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.618859 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.618816 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11d42b1f-609e-4901-9b31-76a2426fc294-registry-certificates\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.619253 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.619234 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11d42b1f-609e-4901-9b31-76a2426fc294-trusted-ca\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.620510 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.620484 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11d42b1f-609e-4901-9b31-76a2426fc294-registry-tls\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.620700 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.620683 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11d42b1f-609e-4901-9b31-76a2426fc294-image-registry-private-configuration\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.620788 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.620772 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11d42b1f-609e-4901-9b31-76a2426fc294-installation-pull-secrets\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.625737 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.625719 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-rkbjf" Oct 30 21:44:41.680953 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.680929 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvmkq\" (UniqueName: \"kubernetes.io/projected/11d42b1f-609e-4901-9b31-76a2426fc294-kube-api-access-vvmkq\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.681107 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.681083 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11d42b1f-609e-4901-9b31-76a2426fc294-bound-sa-token\") pod \"image-registry-56fd84f888-knnp4\" (UID: \"11d42b1f-609e-4901-9b31-76a2426fc294\") " pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:41.729440 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:41.729416 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:42.040330 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:42.040270 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-rkbjf"] Oct 30 21:44:42.043626 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:42.043589 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9274080_66f5_4497_8e0d_1bb850db463e.slice/crio-f4b4a4bf49c9b89d95154f5c0715911ddcfea43067f57dba339343f783220bd1 WatchSource:0}: Error finding container f4b4a4bf49c9b89d95154f5c0715911ddcfea43067f57dba339343f783220bd1: Status 404 returned error can't find the container with id f4b4a4bf49c9b89d95154f5c0715911ddcfea43067f57dba339343f783220bd1 Oct 30 21:44:42.098459 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:42.098439 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56fd84f888-knnp4"] Oct 30 21:44:42.101030 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:42.101009 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d42b1f_609e_4901_9b31_76a2426fc294.slice/crio-177cb87bd4133e81bf23629cf97aed823ead0befcd29e8f1c635bf4635aaf797 WatchSource:0}: Error finding container 177cb87bd4133e81bf23629cf97aed823ead0befcd29e8f1c635bf4635aaf797: Status 404 returned error can't find the container with id 177cb87bd4133e81bf23629cf97aed823ead0befcd29e8f1c635bf4635aaf797 Oct 30 21:44:42.167624 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:42.167598 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sqfth" Oct 30 21:44:42.196412 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:42.196386 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56fd84f888-knnp4" event={"ID":"11d42b1f-609e-4901-9b31-76a2426fc294","Type":"ContainerStarted","Data":"4f9432777fa9f24dab26cd40ea47545c94b3fc1f0787917bc26eb0d202c56051"} Oct 30 21:44:42.196539 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:42.196432 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:44:42.196539 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:42.196453 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56fd84f888-knnp4" event={"ID":"11d42b1f-609e-4901-9b31-76a2426fc294","Type":"ContainerStarted","Data":"177cb87bd4133e81bf23629cf97aed823ead0befcd29e8f1c635bf4635aaf797"} Oct 30 21:44:42.197734 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:42.197713 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-rkbjf" event={"ID":"b9274080-66f5-4497-8e0d-1bb850db463e","Type":"ContainerStarted","Data":"f4b4a4bf49c9b89d95154f5c0715911ddcfea43067f57dba339343f783220bd1"} Oct 30 21:44:42.403876 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:42.403815 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56fd84f888-knnp4" podStartSLOduration=1.403798045 podStartE2EDuration="1.403798045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:42.403232438 +0000 UTC m=+15.503664177" watchObservedRunningTime="2026-10-30 21:44:42.403798045 +0000 UTC m=+15.504229775" Oct 30 21:44:45.146347 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:45.146314 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t268b"] Oct 30 21:44:45.148398 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:45.148372 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t268b" Oct 30 21:44:45.233524 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:45.233496 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-j7wnb\"" Oct 30 21:44:45.249991 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:45.249951 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 21:44:45.250676 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:45.250656 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t268b"] Oct 30 21:44:45.251194 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:45.251162 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/80badb63-a1f9-437a-aeb0-d0ca556b99e0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-t268b\" (UID: \"80badb63-a1f9-437a-aeb0-d0ca556b99e0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t268b" Oct 30 21:44:45.351535 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:45.351500 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/80badb63-a1f9-437a-aeb0-d0ca556b99e0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-t268b\" (UID: \"80badb63-a1f9-437a-aeb0-d0ca556b99e0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t268b" Oct 30 21:44:45.354286 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:45.354258 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/80badb63-a1f9-437a-aeb0-d0ca556b99e0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-t268b\" (UID: \"80badb63-a1f9-437a-aeb0-d0ca556b99e0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t268b" Oct 30 21:44:45.460175 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:45.460147 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t268b" Oct 30 21:44:45.836018 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:45.835989 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t268b"] Oct 30 21:44:45.839811 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:45.839778 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80badb63_a1f9_437a_aeb0_d0ca556b99e0.slice/crio-1cbe2cffb75c53f7e37f4da9eed496aac0dc184702b1159a2b36b7250626c079 WatchSource:0}: Error finding container 1cbe2cffb75c53f7e37f4da9eed496aac0dc184702b1159a2b36b7250626c079: Status 404 returned error can't find the container with id 1cbe2cffb75c53f7e37f4da9eed496aac0dc184702b1159a2b36b7250626c079 Oct 30 21:44:46.211101 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.211019 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-9875ddfb4-b4qrt"] Oct 30 21:44:46.213670 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.213642 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.215950 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.215929 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t268b" event={"ID":"80badb63-a1f9-437a-aeb0-d0ca556b99e0","Type":"ContainerStarted","Data":"1cbe2cffb75c53f7e37f4da9eed496aac0dc184702b1159a2b36b7250626c079"} Oct 30 21:44:46.259206 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.259183 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-oauth-serving-cert\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.259310 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.259266 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e69fc3e3-a16c-45b9-b108-e6f71b91d703-tmp\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.259374 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.259317 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-config\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.259425 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.259393 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckxb9\" (UniqueName: \"kubernetes.io/projected/e69fc3e3-a16c-45b9-b108-e6f71b91d703-kube-api-access-ckxb9\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.259475 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.259431 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-oauth-config\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.259475 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.259459 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-service-ca\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.259569 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.259507 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-serving-cert\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.319203 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.319166 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 21:44:46.319320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.319204 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 21:44:46.319320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.319295 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 21:44:46.319441 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.319364 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 21:44:46.319441 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.319384 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 21:44:46.319527 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.319448 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-77fbt\"" Oct 30 21:44:46.320132 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.320116 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9875ddfb4-b4qrt"] Oct 30 21:44:46.360693 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.360668 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-config\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.360811 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.360710 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckxb9\" (UniqueName: \"kubernetes.io/projected/e69fc3e3-a16c-45b9-b108-e6f71b91d703-kube-api-access-ckxb9\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.360811 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.360743 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-oauth-config\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.360919 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.360876 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-service-ca\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.360974 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.360938 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-serving-cert\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.361027 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.360990 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-oauth-serving-cert\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.361027 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.361016 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e69fc3e3-a16c-45b9-b108-e6f71b91d703-tmp\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.361359 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.361339 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e69fc3e3-a16c-45b9-b108-e6f71b91d703-tmp\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.361507 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.361483 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-config\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.361554 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.361522 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-service-ca\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.361832 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.361807 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-oauth-serving-cert\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.363710 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.363690 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-serving-cert\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.363999 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.363980 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-oauth-config\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.421585 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.421558 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckxb9\" (UniqueName: \"kubernetes.io/projected/e69fc3e3-a16c-45b9-b108-e6f71b91d703-kube-api-access-ckxb9\") pod \"console-9875ddfb4-b4qrt\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.524982 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.524929 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:46.901605 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:46.901579 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9875ddfb4-b4qrt"] Oct 30 21:44:46.906479 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:46.906450 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69fc3e3_a16c_45b9_b108_e6f71b91d703.slice/crio-6fad0b82e388b829febdd311b857c903a2fe098481c2ea95a74227cfd1d1d45c WatchSource:0}: Error finding container 6fad0b82e388b829febdd311b857c903a2fe098481c2ea95a74227cfd1d1d45c: Status 404 returned error can't find the container with id 6fad0b82e388b829febdd311b857c903a2fe098481c2ea95a74227cfd1d1d45c Oct 30 21:44:47.220677 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:47.220590 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9875ddfb4-b4qrt" event={"ID":"e69fc3e3-a16c-45b9-b108-e6f71b91d703","Type":"ContainerStarted","Data":"6fad0b82e388b829febdd311b857c903a2fe098481c2ea95a74227cfd1d1d45c"} Oct 30 21:44:48.230940 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:48.230890 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t268b" event={"ID":"80badb63-a1f9-437a-aeb0-d0ca556b99e0","Type":"ContainerStarted","Data":"9b226b5e222004089110d86fdc37602a936285fd0244d315ad7bf1782b36644b"} Oct 30 21:44:48.231368 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:48.231172 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t268b" Oct 30 21:44:48.236499 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:48.236480 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t268b" Oct 30 21:44:48.463722 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:48.463659 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t268b" podStartSLOduration=1.892423366 podStartE2EDuration="3.463640703s" totalImagesPullingTime="1.571217337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:48.346004331 +0000 UTC m=+21.446436062" watchObservedRunningTime="2026-10-30 21:44:48.463640703 +0000 UTC m=+21.564072433" Oct 30 21:44:51.244145 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:51.244113 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9875ddfb4-b4qrt" event={"ID":"e69fc3e3-a16c-45b9-b108-e6f71b91d703","Type":"ContainerStarted","Data":"96d50f88fff4ed30bd9f96ee549b579863f0babad0a04a215cd93a1bc5c77069"} Oct 30 21:44:51.361524 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:51.361466 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-9875ddfb4-b4qrt" podStartSLOduration=2.026612172 podStartE2EDuration="5.361447967s" totalImagesPullingTime="3.334835795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:51.36100402 +0000 UTC m=+24.461435742" watchObservedRunningTime="2026-10-30 21:44:51.361447967 +0000 UTC m=+24.461879702" Oct 30 21:44:55.321996 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:44:55.321945 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:44:56.323602 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.323572 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z7ndn"] Oct 30 21:44:56.353646 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.353602 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.409875 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.409848 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:44:56.433839 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.433809 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e295034c-effb-4a8d-ad28-d2068c6c1616-root\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.433979 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.433865 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e295034c-effb-4a8d-ad28-d2068c6c1616-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.433979 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.433898 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e295034c-effb-4a8d-ad28-d2068c6c1616-node-exporter-textfile\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.433979 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.433932 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e295034c-effb-4a8d-ad28-d2068c6c1616-node-exporter-accelerators-collector-config\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.433979 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.433967 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e295034c-effb-4a8d-ad28-d2068c6c1616-sys\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.434183 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.433988 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e295034c-effb-4a8d-ad28-d2068c6c1616-node-exporter-tls\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.434183 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.434019 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e295034c-effb-4a8d-ad28-d2068c6c1616-metrics-client-ca\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.434183 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.434053 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zb8r\" (UniqueName: \"kubernetes.io/projected/e295034c-effb-4a8d-ad28-d2068c6c1616-kube-api-access-4zb8r\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.434183 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.434082 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e295034c-effb-4a8d-ad28-d2068c6c1616-node-exporter-wtmp\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.459004 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.458982 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:44:56.459145 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.459133 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7g595\"" Oct 30 21:44:56.459253 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.459237 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:44:56.459397 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.459276 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:44:56.459397 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.459285 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:44:56.459397 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.459339 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:44:56.525884 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.525858 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:56.526574 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.526557 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:56.532855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.532836 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:56.535166 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.535139 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e295034c-effb-4a8d-ad28-d2068c6c1616-node-exporter-tls\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.535268 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.535173 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e295034c-effb-4a8d-ad28-d2068c6c1616-metrics-client-ca\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.535268 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.535202 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zb8r\" (UniqueName: \"kubernetes.io/projected/e295034c-effb-4a8d-ad28-d2068c6c1616-kube-api-access-4zb8r\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.535375 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.535357 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e295034c-effb-4a8d-ad28-d2068c6c1616-node-exporter-wtmp\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.535424 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.535404 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e295034c-effb-4a8d-ad28-d2068c6c1616-root\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.535488 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.535453 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e295034c-effb-4a8d-ad28-d2068c6c1616-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.535553 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.535486 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e295034c-effb-4a8d-ad28-d2068c6c1616-node-exporter-textfile\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.535553 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.535518 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e295034c-effb-4a8d-ad28-d2068c6c1616-root\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.535553 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.535521 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e295034c-effb-4a8d-ad28-d2068c6c1616-node-exporter-accelerators-collector-config\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.535553 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.535530 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e295034c-effb-4a8d-ad28-d2068c6c1616-node-exporter-wtmp\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.535783 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.535588 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e295034c-effb-4a8d-ad28-d2068c6c1616-sys\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.535783 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.535665 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e295034c-effb-4a8d-ad28-d2068c6c1616-sys\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.535886 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.535849 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e295034c-effb-4a8d-ad28-d2068c6c1616-node-exporter-textfile\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.535934 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.535901 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e295034c-effb-4a8d-ad28-d2068c6c1616-metrics-client-ca\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.536061 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.535996 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e295034c-effb-4a8d-ad28-d2068c6c1616-node-exporter-accelerators-collector-config\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.537953 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.537932 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e295034c-effb-4a8d-ad28-d2068c6c1616-node-exporter-tls\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.544970 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.544940 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e295034c-effb-4a8d-ad28-d2068c6c1616-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.595452 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.595399 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zb8r\" (UniqueName: \"kubernetes.io/projected/e295034c-effb-4a8d-ad28-d2068c6c1616-kube-api-access-4zb8r\") pod \"node-exporter-z7ndn\" (UID: \"e295034c-effb-4a8d-ad28-d2068c6c1616\") " pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.706658 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:56.706608 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z7ndn" Oct 30 21:44:56.740833 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:56.740804 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode295034c_effb_4a8d_ad28_d2068c6c1616.slice/crio-1f29a7da17ed48e0e760f426ddb73b54fdd0522372217328e9f75db64db83cd5 WatchSource:0}: Error finding container 1f29a7da17ed48e0e760f426ddb73b54fdd0522372217328e9f75db64db83cd5: Status 404 returned error can't find the container with id 1f29a7da17ed48e0e760f426ddb73b54fdd0522372217328e9f75db64db83cd5 Oct 30 21:44:57.265814 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:57.265772 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z7ndn" event={"ID":"e295034c-effb-4a8d-ad28-d2068c6c1616","Type":"ContainerStarted","Data":"1f29a7da17ed48e0e760f426ddb73b54fdd0522372217328e9f75db64db83cd5"} Oct 30 21:44:57.270281 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:57.270259 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:44:58.735870 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.735844 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-776dd8d5bd-b52b7"] Oct 30 21:44:58.749480 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.749460 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.803102 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.803073 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-776dd8d5bd-b52b7"] Oct 30 21:44:58.854442 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.854417 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-config\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.854553 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.854448 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-service-ca\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.854553 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.854468 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-serving-cert\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.854725 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.854556 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6clsf\" (UniqueName: \"kubernetes.io/projected/db072ad6-3e9e-4695-9a94-c1d976d3af99-kube-api-access-6clsf\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.854781 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.854746 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db072ad6-3e9e-4695-9a94-c1d976d3af99-tmp\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.854890 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.854794 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-oauth-config\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.854890 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.854816 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-oauth-serving-cert\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.854890 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.854867 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-trusted-ca-bundle\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.948811 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.948750 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 21:44:58.955480 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.955456 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6clsf\" (UniqueName: \"kubernetes.io/projected/db072ad6-3e9e-4695-9a94-c1d976d3af99-kube-api-access-6clsf\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.955586 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.955510 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db072ad6-3e9e-4695-9a94-c1d976d3af99-tmp\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.955586 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.955555 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-oauth-config\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.955586 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.955580 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-oauth-serving-cert\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.955840 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.955642 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-trusted-ca-bundle\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.955840 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.955670 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-config\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.955840 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.955691 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-service-ca\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.955840 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.955721 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-serving-cert\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.956047 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.955943 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db072ad6-3e9e-4695-9a94-c1d976d3af99-tmp\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.956537 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.956491 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-oauth-serving-cert\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.956668 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.956571 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-service-ca\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.956668 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.956594 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-trusted-ca-bundle\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.956799 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.956777 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-config\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.958163 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.958140 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-oauth-config\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:58.958270 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:58.958254 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-serving-cert\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:59.022299 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:59.022280 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6clsf\" (UniqueName: \"kubernetes.io/projected/db072ad6-3e9e-4695-9a94-c1d976d3af99-kube-api-access-6clsf\") pod \"console-776dd8d5bd-b52b7\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:59.075477 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:59.075450 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:44:59.273361 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:59.273307 4335 generic.go:356] "Generic (PLEG): container finished" podID="e295034c-effb-4a8d-ad28-d2068c6c1616" containerID="59e79e05e2233c92e3b2c5c937712f9d4f910e28f2cf00c75ef01bf98c3c888c" exitCode=0 Oct 30 21:44:59.273482 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:59.273384 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z7ndn" event={"ID":"e295034c-effb-4a8d-ad28-d2068c6c1616","Type":"ContainerDied","Data":"59e79e05e2233c92e3b2c5c937712f9d4f910e28f2cf00c75ef01bf98c3c888c"} Oct 30 21:44:59.507958 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:44:59.507929 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-776dd8d5bd-b52b7"] Oct 30 21:44:59.511332 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:44:59.511309 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb072ad6_3e9e_4695_9a94_c1d976d3af99.slice/crio-8867a712b81659309566144e8c7a9255800d74f8a69a434047e3fa84c061f21c WatchSource:0}: Error finding container 8867a712b81659309566144e8c7a9255800d74f8a69a434047e3fa84c061f21c: Status 404 returned error can't find the container with id 8867a712b81659309566144e8c7a9255800d74f8a69a434047e3fa84c061f21c Oct 30 21:45:00.008425 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:00.008395 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tcckj" Oct 30 21:45:00.280154 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:00.280046 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776dd8d5bd-b52b7" event={"ID":"db072ad6-3e9e-4695-9a94-c1d976d3af99","Type":"ContainerStarted","Data":"bd84f172d856295ca786bab4455f8998d1715a7c7e0697bf40a5837e98c1a978"} Oct 30 21:45:00.280154 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:00.280128 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776dd8d5bd-b52b7" event={"ID":"db072ad6-3e9e-4695-9a94-c1d976d3af99","Type":"ContainerStarted","Data":"8867a712b81659309566144e8c7a9255800d74f8a69a434047e3fa84c061f21c"} Oct 30 21:45:00.283054 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:00.283022 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-rkbjf" event={"ID":"b9274080-66f5-4497-8e0d-1bb850db463e","Type":"ContainerStarted","Data":"64ee0aea7843dccc10db885f2bd40002de7b64f8cbf04043d6879aad5c2b27a9"} Oct 30 21:45:00.283679 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:00.283658 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-rkbjf" Oct 30 21:45:00.286589 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:00.286569 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z7ndn" event={"ID":"e295034c-effb-4a8d-ad28-d2068c6c1616","Type":"ContainerStarted","Data":"51bb59c075ce8f172a3a59186035e4d516a0eea024b509464b5cee3a2e02e027"} Oct 30 21:45:00.286705 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:00.286604 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z7ndn" event={"ID":"e295034c-effb-4a8d-ad28-d2068c6c1616","Type":"ContainerStarted","Data":"4a0c80d7a67aa2acacd18b149761f94a652cd25d2fd4d52115a121555c5ed424"} Oct 30 21:45:00.296481 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:00.296454 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-rkbjf" Oct 30 21:45:00.397838 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:00.397782 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-776dd8d5bd-b52b7" podStartSLOduration=2.397763548 podStartE2EDuration="2.397763548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:00.397105556 +0000 UTC m=+33.497537287" watchObservedRunningTime="2026-10-30 21:45:00.397763548 +0000 UTC m=+33.498195284" Oct 30 21:45:00.516481 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:00.516416 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z7ndn" podStartSLOduration=2.54706654 podStartE2EDuration="4.516397778s" totalImagesPullingTime="1.877702647s" totalInitContainerRuntime="91.628591ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:44:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:00.51574293 +0000 UTC m=+33.616174662" watchObservedRunningTime="2026-10-30 21:45:00.516397778 +0000 UTC m=+33.616829512" Oct 30 21:45:00.637584 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:00.637526 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-rkbjf" podStartSLOduration=2.332845786 podStartE2EDuration="19.637509092s" totalImagesPullingTime="17.304663306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:00.637493545 +0000 UTC m=+33.737925269" watchObservedRunningTime="2026-10-30 21:45:00.637509092 +0000 UTC m=+33.737940826" Oct 30 21:45:03.206795 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:03.206769 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56fd84f888-knnp4" Oct 30 21:45:08.975803 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:08.975659 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-776dd8d5bd-b52b7"] Oct 30 21:45:09.023489 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.023458 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-d55cbfcd-q47vs"] Oct 30 21:45:09.070154 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.070127 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.075635 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.075599 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:45:09.087051 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.087029 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d55cbfcd-q47vs"] Oct 30 21:45:09.231901 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.231853 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-oauth-config\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.231901 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.231881 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-trusted-ca-bundle\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.231901 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.231897 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-oauth-serving-cert\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.232222 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.231918 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-service-ca\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.232222 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.232020 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzdv2\" (UniqueName: \"kubernetes.io/projected/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-kube-api-access-kzdv2\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.232222 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.232082 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-tmp\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.232222 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.232135 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-serving-cert\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.232222 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.232201 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-config\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.333254 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.333231 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-tmp\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.333368 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.333257 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-serving-cert\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.333368 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.333276 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-config\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.333368 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.333317 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-oauth-config\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.333482 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.333462 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-trusted-ca-bundle\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.333523 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.333507 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-oauth-serving-cert\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.333568 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.333553 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-service-ca\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.333610 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.333592 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzdv2\" (UniqueName: \"kubernetes.io/projected/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-kube-api-access-kzdv2\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.333681 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.333670 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-tmp\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.334127 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.334099 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-config\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.334241 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.334223 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-oauth-serving-cert\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.334360 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.334336 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-service-ca\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.334593 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.334575 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-trusted-ca-bundle\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.336522 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.336502 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-oauth-config\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.336733 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.336713 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-serving-cert\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.394497 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.394472 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzdv2\" (UniqueName: \"kubernetes.io/projected/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-kube-api-access-kzdv2\") pod \"console-d55cbfcd-q47vs\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:09.692958 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:09.692909 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:10.104065 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:10.104038 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d55cbfcd-q47vs"] Oct 30 21:45:10.127405 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:45:10.127381 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ac4a15_4f8b_402e_a043_cd0e6b9c55e1.slice/crio-1e9a31a61af8e775f351c9132b8a8d84ae4065b0a1fb744b791929b84e89259b WatchSource:0}: Error finding container 1e9a31a61af8e775f351c9132b8a8d84ae4065b0a1fb744b791929b84e89259b: Status 404 returned error can't find the container with id 1e9a31a61af8e775f351c9132b8a8d84ae4065b0a1fb744b791929b84e89259b Oct 30 21:45:10.324675 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:10.324643 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d55cbfcd-q47vs" event={"ID":"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1","Type":"ContainerStarted","Data":"f8de0b87092c4d94567e4633296c8b3d2249ef645c87bcd3f3b7b9c47207722d"} Oct 30 21:45:10.324866 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:10.324696 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d55cbfcd-q47vs" event={"ID":"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1","Type":"ContainerStarted","Data":"1e9a31a61af8e775f351c9132b8a8d84ae4065b0a1fb744b791929b84e89259b"} Oct 30 21:45:10.442160 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:10.442074 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-d55cbfcd-q47vs" podStartSLOduration=2.442056657 podStartE2EDuration="2.442056657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:10.441450221 +0000 UTC m=+43.541881954" watchObservedRunningTime="2026-10-30 21:45:10.442056657 +0000 UTC m=+43.542488390" Oct 30 21:45:19.693757 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:19.693713 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:19.694427 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:19.693783 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:19.698660 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:19.698633 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:20.363444 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:20.363406 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:45:20.531880 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:20.531847 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9875ddfb4-b4qrt"] Oct 30 21:45:25.321263 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:45:25.321214 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:45:28.712512 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.712479 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d7444f6d7-2c5tz"] Oct 30 21:45:28.743759 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.743725 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.774831 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.774807 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d7444f6d7-2c5tz"] Oct 30 21:45:28.871795 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.871772 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4gn9\" (UniqueName: \"kubernetes.io/projected/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-kube-api-access-z4gn9\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.871910 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.871803 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-console-oauth-config\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.871910 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.871824 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-console-config\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.871910 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.871844 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-oauth-serving-cert\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.872072 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.871972 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-service-ca\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.872072 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.872015 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-console-serving-cert\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.872175 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.872105 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-trusted-ca-bundle\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.872175 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.872131 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-tmp\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.973041 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.972982 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-console-oauth-config\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.973041 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.973017 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-console-config\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.973198 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.973044 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-oauth-serving-cert\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.973198 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.973075 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-service-ca\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.973198 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.973102 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-console-serving-cert\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.973198 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.973143 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-trusted-ca-bundle\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.973198 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.973167 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-tmp\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.973446 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.973234 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4gn9\" (UniqueName: \"kubernetes.io/projected/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-kube-api-access-z4gn9\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.973589 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.973549 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-tmp\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.974017 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.973997 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-console-config\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.974126 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.974106 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-oauth-serving-cert\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.974205 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.974182 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-service-ca\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.974338 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.974317 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-trusted-ca-bundle\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.975741 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.975723 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-console-serving-cert\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:28.975833 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:28.975729 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-console-oauth-config\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:29.034406 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:29.034380 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4gn9\" (UniqueName: \"kubernetes.io/projected/8dc9d04e-4582-43d8-9d54-0f7d61d27daf-kube-api-access-z4gn9\") pod \"console-5d7444f6d7-2c5tz\" (UID: \"8dc9d04e-4582-43d8-9d54-0f7d61d27daf\") " pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:29.065937 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:29.065919 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:29.463735 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:29.463671 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d7444f6d7-2c5tz"] Oct 30 21:45:29.466189 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:45:29.466161 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc9d04e_4582_43d8_9d54_0f7d61d27daf.slice/crio-7e3c37ca061d5fdfb1ae2a46489526dcf016adcb1f3d34c89fae497cb09e228e WatchSource:0}: Error finding container 7e3c37ca061d5fdfb1ae2a46489526dcf016adcb1f3d34c89fae497cb09e228e: Status 404 returned error can't find the container with id 7e3c37ca061d5fdfb1ae2a46489526dcf016adcb1f3d34c89fae497cb09e228e Oct 30 21:45:30.393889 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:30.393851 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7444f6d7-2c5tz" event={"ID":"8dc9d04e-4582-43d8-9d54-0f7d61d27daf","Type":"ContainerStarted","Data":"bb743e808c7f74c6da4d15777f49c22688c868a71ddf8f923bf9109e87eb45cd"} Oct 30 21:45:30.394413 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:30.393911 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7444f6d7-2c5tz" event={"ID":"8dc9d04e-4582-43d8-9d54-0f7d61d27daf","Type":"ContainerStarted","Data":"7e3c37ca061d5fdfb1ae2a46489526dcf016adcb1f3d34c89fae497cb09e228e"} Oct 30 21:45:30.560849 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:30.560799 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5d7444f6d7-2c5tz" podStartSLOduration=2.560783345 podStartE2EDuration="2.560783345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:30.510922373 +0000 UTC m=+63.611354104" watchObservedRunningTime="2026-10-30 21:45:30.560783345 +0000 UTC m=+63.661215078" Oct 30 21:45:33.442310 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:33.442278 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-bcs5k-f5hhp"] Oct 30 21:45:33.442858 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:33.442496 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" podUID="d45916e8-c498-4bf5-b5f3-df697b608ed9" containerName="insights-gathering" containerID="cri-o://3af588a860ebfaa70bb64808d17d784b73849d688080c5d46ecb233ac04fdeb2" gracePeriod=30 Oct 30 21:45:33.731221 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:33.731200 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:45:33.805475 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:33.805454 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d45916e8-c498-4bf5-b5f3-df697b608ed9-kube-api-access-lrgmx\" (UniqueName: \"kubernetes.io/projected/d45916e8-c498-4bf5-b5f3-df697b608ed9-kube-api-access-lrgmx\") pod \"d45916e8-c498-4bf5-b5f3-df697b608ed9\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " Oct 30 21:45:33.805581 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:33.805493 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d45916e8-c498-4bf5-b5f3-df697b608ed9-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d45916e8-c498-4bf5-b5f3-df697b608ed9-archives-path\") pod \"d45916e8-c498-4bf5-b5f3-df697b608ed9\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " Oct 30 21:45:33.805581 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:33.805540 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-service-ca-bundle\") pod \"d45916e8-c498-4bf5-b5f3-df697b608ed9\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " Oct 30 21:45:33.805581 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:33.805562 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-trusted-ca-bundle\") pod \"d45916e8-c498-4bf5-b5f3-df697b608ed9\" (UID: \"d45916e8-c498-4bf5-b5f3-df697b608ed9\") " Oct 30 21:45:33.805867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:33.805845 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45916e8-c498-4bf5-b5f3-df697b608ed9-archives-path" pod "d45916e8-c498-4bf5-b5f3-df697b608ed9" (UID: "d45916e8-c498-4bf5-b5f3-df697b608ed9"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:33.805962 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:33.805941 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-service-ca-bundle" pod "d45916e8-c498-4bf5-b5f3-df697b608ed9" (UID: "d45916e8-c498-4bf5-b5f3-df697b608ed9"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:33.806032 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:33.806013 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-trusted-ca-bundle" pod "d45916e8-c498-4bf5-b5f3-df697b608ed9" (UID: "d45916e8-c498-4bf5-b5f3-df697b608ed9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:33.807725 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:33.807705 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45916e8-c498-4bf5-b5f3-df697b608ed9-kube-api-access-lrgmx" pod "d45916e8-c498-4bf5-b5f3-df697b608ed9" (UID: "d45916e8-c498-4bf5-b5f3-df697b608ed9"). InnerVolumeSpecName "kube-api-access-lrgmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:33.906693 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:33.906669 4335 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-service-ca-bundle\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:33.906693 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:33.906690 4335 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d45916e8-c498-4bf5-b5f3-df697b608ed9-trusted-ca-bundle\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:33.906816 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:33.906701 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrgmx\" (UniqueName: \"kubernetes.io/projected/d45916e8-c498-4bf5-b5f3-df697b608ed9-kube-api-access-lrgmx\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:33.906816 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:33.906711 4335 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d45916e8-c498-4bf5-b5f3-df697b608ed9-archives-path\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:33.997696 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:33.997600 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-776dd8d5bd-b52b7" podUID="db072ad6-3e9e-4695-9a94-c1d976d3af99" containerName="console" containerID="cri-o://bd84f172d856295ca786bab4455f8998d1715a7c7e0697bf40a5837e98c1a978" gracePeriod=15 Oct 30 21:45:34.271409 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.271359 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:45:34.309190 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.309163 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-config\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-config\") pod \"db072ad6-3e9e-4695-9a94-c1d976d3af99\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " Oct 30 21:45:34.309299 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.309198 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-service-ca\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-service-ca\") pod \"db072ad6-3e9e-4695-9a94-c1d976d3af99\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " Oct 30 21:45:34.309299 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.309219 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db072ad6-3e9e-4695-9a94-c1d976d3af99-kube-api-access-6clsf\" (UniqueName: \"kubernetes.io/projected/db072ad6-3e9e-4695-9a94-c1d976d3af99-kube-api-access-6clsf\") pod \"db072ad6-3e9e-4695-9a94-c1d976d3af99\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " Oct 30 21:45:34.309299 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.309240 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db072ad6-3e9e-4695-9a94-c1d976d3af99-tmp\" (UniqueName: \"kubernetes.io/empty-dir/db072ad6-3e9e-4695-9a94-c1d976d3af99-tmp\") pod \"db072ad6-3e9e-4695-9a94-c1d976d3af99\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " Oct 30 21:45:34.309299 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.309266 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-serving-cert\") pod \"db072ad6-3e9e-4695-9a94-c1d976d3af99\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " Oct 30 21:45:34.309472 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.309311 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-oauth-config\") pod \"db072ad6-3e9e-4695-9a94-c1d976d3af99\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " Oct 30 21:45:34.309472 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.309338 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-oauth-serving-cert\") pod \"db072ad6-3e9e-4695-9a94-c1d976d3af99\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " Oct 30 21:45:34.309472 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.309364 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-trusted-ca-bundle\") pod \"db072ad6-3e9e-4695-9a94-c1d976d3af99\" (UID: \"db072ad6-3e9e-4695-9a94-c1d976d3af99\") " Oct 30 21:45:34.309644 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.309574 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db072ad6-3e9e-4695-9a94-c1d976d3af99-tmp" pod "db072ad6-3e9e-4695-9a94-c1d976d3af99" (UID: "db072ad6-3e9e-4695-9a94-c1d976d3af99"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:34.309708 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.309678 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-config" pod "db072ad6-3e9e-4695-9a94-c1d976d3af99" (UID: "db072ad6-3e9e-4695-9a94-c1d976d3af99"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:34.309708 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.309683 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-service-ca" pod "db072ad6-3e9e-4695-9a94-c1d976d3af99" (UID: "db072ad6-3e9e-4695-9a94-c1d976d3af99"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:34.309912 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.309890 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-trusted-ca-bundle" pod "db072ad6-3e9e-4695-9a94-c1d976d3af99" (UID: "db072ad6-3e9e-4695-9a94-c1d976d3af99"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:34.310013 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.309990 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-oauth-serving-cert" pod "db072ad6-3e9e-4695-9a94-c1d976d3af99" (UID: "db072ad6-3e9e-4695-9a94-c1d976d3af99"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:34.311569 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.311537 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db072ad6-3e9e-4695-9a94-c1d976d3af99-kube-api-access-6clsf" pod "db072ad6-3e9e-4695-9a94-c1d976d3af99" (UID: "db072ad6-3e9e-4695-9a94-c1d976d3af99"). InnerVolumeSpecName "kube-api-access-6clsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:34.311675 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.311575 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-serving-cert" pod "db072ad6-3e9e-4695-9a94-c1d976d3af99" (UID: "db072ad6-3e9e-4695-9a94-c1d976d3af99"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:34.311675 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.311608 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-oauth-config" pod "db072ad6-3e9e-4695-9a94-c1d976d3af99" (UID: "db072ad6-3e9e-4695-9a94-c1d976d3af99"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:34.408742 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.408703 4335 generic.go:356] "Generic (PLEG): container finished" podID="db072ad6-3e9e-4695-9a94-c1d976d3af99" containerID="bd84f172d856295ca786bab4455f8998d1715a7c7e0697bf40a5837e98c1a978" exitCode=0 Oct 30 21:45:34.408841 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.408765 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776dd8d5bd-b52b7" Oct 30 21:45:34.408841 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.408794 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776dd8d5bd-b52b7" event={"ID":"db072ad6-3e9e-4695-9a94-c1d976d3af99","Type":"ContainerDied","Data":"bd84f172d856295ca786bab4455f8998d1715a7c7e0697bf40a5837e98c1a978"} Oct 30 21:45:34.408946 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.408845 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776dd8d5bd-b52b7" event={"ID":"db072ad6-3e9e-4695-9a94-c1d976d3af99","Type":"ContainerDied","Data":"8867a712b81659309566144e8c7a9255800d74f8a69a434047e3fa84c061f21c"} Oct 30 21:45:34.408946 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.408867 4335 scope.go:122] "RemoveContainer" containerID="bd84f172d856295ca786bab4455f8998d1715a7c7e0697bf40a5837e98c1a978" Oct 30 21:45:34.409903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.409877 4335 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-oauth-config\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:34.410141 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.409906 4335 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-oauth-serving-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:34.410141 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.409920 4335 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-trusted-ca-bundle\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:34.410141 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.409935 4335 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-config\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:34.410141 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.409950 4335 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db072ad6-3e9e-4695-9a94-c1d976d3af99-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:34.410141 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.409964 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6clsf\" (UniqueName: \"kubernetes.io/projected/db072ad6-3e9e-4695-9a94-c1d976d3af99-kube-api-access-6clsf\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:34.410141 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.409973 4335 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db072ad6-3e9e-4695-9a94-c1d976d3af99-tmp\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:34.410141 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.409983 4335 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db072ad6-3e9e-4695-9a94-c1d976d3af99-console-serving-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:34.410505 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.410193 4335 generic.go:356] "Generic (PLEG): container finished" podID="d45916e8-c498-4bf5-b5f3-df697b608ed9" containerID="3af588a860ebfaa70bb64808d17d784b73849d688080c5d46ecb233ac04fdeb2" exitCode=2 Oct 30 21:45:34.410505 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.410242 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" Oct 30 21:45:34.410505 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.410251 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" event={"ID":"d45916e8-c498-4bf5-b5f3-df697b608ed9","Type":"ContainerDied","Data":"3af588a860ebfaa70bb64808d17d784b73849d688080c5d46ecb233ac04fdeb2"} Oct 30 21:45:34.410505 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.410286 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bcs5k-f5hhp" event={"ID":"d45916e8-c498-4bf5-b5f3-df697b608ed9","Type":"ContainerDied","Data":"55c67840b3a6b6d25f6059d282483d6436754966dd76cadfbab8fba03c0c5f16"} Oct 30 21:45:34.417848 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.417829 4335 scope.go:122] "RemoveContainer" containerID="bd84f172d856295ca786bab4455f8998d1715a7c7e0697bf40a5837e98c1a978" Oct 30 21:45:34.418109 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:45:34.418087 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd84f172d856295ca786bab4455f8998d1715a7c7e0697bf40a5837e98c1a978\": container with ID starting with bd84f172d856295ca786bab4455f8998d1715a7c7e0697bf40a5837e98c1a978 not found: ID does not exist" containerID="bd84f172d856295ca786bab4455f8998d1715a7c7e0697bf40a5837e98c1a978" Oct 30 21:45:34.418208 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.418120 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd84f172d856295ca786bab4455f8998d1715a7c7e0697bf40a5837e98c1a978"} err="failed to get container status \"bd84f172d856295ca786bab4455f8998d1715a7c7e0697bf40a5837e98c1a978\": rpc error: code = NotFound desc = could not find container \"bd84f172d856295ca786bab4455f8998d1715a7c7e0697bf40a5837e98c1a978\": container with ID starting with bd84f172d856295ca786bab4455f8998d1715a7c7e0697bf40a5837e98c1a978 not found: ID does not exist" Oct 30 21:45:34.418208 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.418161 4335 scope.go:122] "RemoveContainer" containerID="3af588a860ebfaa70bb64808d17d784b73849d688080c5d46ecb233ac04fdeb2" Oct 30 21:45:34.425346 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.425329 4335 scope.go:122] "RemoveContainer" containerID="3af588a860ebfaa70bb64808d17d784b73849d688080c5d46ecb233ac04fdeb2" Oct 30 21:45:34.425595 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:45:34.425561 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af588a860ebfaa70bb64808d17d784b73849d688080c5d46ecb233ac04fdeb2\": container with ID starting with 3af588a860ebfaa70bb64808d17d784b73849d688080c5d46ecb233ac04fdeb2 not found: ID does not exist" containerID="3af588a860ebfaa70bb64808d17d784b73849d688080c5d46ecb233ac04fdeb2" Oct 30 21:45:34.425662 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.425603 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af588a860ebfaa70bb64808d17d784b73849d688080c5d46ecb233ac04fdeb2"} err="failed to get container status \"3af588a860ebfaa70bb64808d17d784b73849d688080c5d46ecb233ac04fdeb2\": rpc error: code = NotFound desc = could not find container \"3af588a860ebfaa70bb64808d17d784b73849d688080c5d46ecb233ac04fdeb2\": container with ID starting with 3af588a860ebfaa70bb64808d17d784b73849d688080c5d46ecb233ac04fdeb2 not found: ID does not exist" Oct 30 21:45:34.573874 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.573849 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-776dd8d5bd-b52b7"] Oct 30 21:45:34.581764 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.581745 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-776dd8d5bd-b52b7"] Oct 30 21:45:34.689513 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.689490 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-bcs5k-f5hhp"] Oct 30 21:45:34.695720 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:34.695694 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-bcs5k-f5hhp"] Oct 30 21:45:35.054346 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:35.054316 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d45916e8-c498-4bf5-b5f3-df697b608ed9" path="/var/lib/kubelet/pods/d45916e8-c498-4bf5-b5f3-df697b608ed9/volumes" Oct 30 21:45:35.054742 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:35.054727 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db072ad6-3e9e-4695-9a94-c1d976d3af99" path="/var/lib/kubelet/pods/db072ad6-3e9e-4695-9a94-c1d976d3af99/volumes" Oct 30 21:45:39.066508 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:39.066465 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:39.067082 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:39.066522 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:39.071227 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:39.071203 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:39.434245 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:39.434170 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d7444f6d7-2c5tz" Oct 30 21:45:39.602787 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:39.602757 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d55cbfcd-q47vs"] Oct 30 21:45:45.551767 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.551693 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-9875ddfb4-b4qrt" podUID="e69fc3e3-a16c-45b9-b108-e6f71b91d703" containerName="console" containerID="cri-o://96d50f88fff4ed30bd9f96ee549b579863f0babad0a04a215cd93a1bc5c77069" gracePeriod=15 Oct 30 21:45:45.843569 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.843539 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:45:45.876908 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.876880 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69fc3e3-a16c-45b9-b108-e6f71b91d703-kube-api-access-ckxb9\" (UniqueName: \"kubernetes.io/projected/e69fc3e3-a16c-45b9-b108-e6f71b91d703-kube-api-access-ckxb9\") pod \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " Oct 30 21:45:45.877016 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.876922 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-config\" (UniqueName: \"kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-config\") pod \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " Oct 30 21:45:45.877016 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.876952 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-service-ca\" (UniqueName: \"kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-service-ca\") pod \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " Oct 30 21:45:45.877134 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.877063 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69fc3e3-a16c-45b9-b108-e6f71b91d703-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e69fc3e3-a16c-45b9-b108-e6f71b91d703-tmp\") pod \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " Oct 30 21:45:45.877134 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.877098 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-oauth-serving-cert\") pod \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " Oct 30 21:45:45.877245 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.877138 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-serving-cert\") pod \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " Oct 30 21:45:45.877309 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.877289 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-oauth-config\") pod \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\" (UID: \"e69fc3e3-a16c-45b9-b108-e6f71b91d703\") " Oct 30 21:45:45.877413 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.877326 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-service-ca" pod "e69fc3e3-a16c-45b9-b108-e6f71b91d703" (UID: "e69fc3e3-a16c-45b9-b108-e6f71b91d703"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:45.877413 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.877374 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-config" pod "e69fc3e3-a16c-45b9-b108-e6f71b91d703" (UID: "e69fc3e3-a16c-45b9-b108-e6f71b91d703"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:45.877413 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.877365 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69fc3e3-a16c-45b9-b108-e6f71b91d703-tmp" pod "e69fc3e3-a16c-45b9-b108-e6f71b91d703" (UID: "e69fc3e3-a16c-45b9-b108-e6f71b91d703"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:45.877590 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.877462 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-oauth-serving-cert" pod "e69fc3e3-a16c-45b9-b108-e6f71b91d703" (UID: "e69fc3e3-a16c-45b9-b108-e6f71b91d703"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:45.877590 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.877531 4335 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-config\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:45.877590 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.877552 4335 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:45.877590 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.877567 4335 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e69fc3e3-a16c-45b9-b108-e6f71b91d703-tmp\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:45.877590 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.877582 4335 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e69fc3e3-a16c-45b9-b108-e6f71b91d703-oauth-serving-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:45.879151 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.879126 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69fc3e3-a16c-45b9-b108-e6f71b91d703-kube-api-access-ckxb9" pod "e69fc3e3-a16c-45b9-b108-e6f71b91d703" (UID: "e69fc3e3-a16c-45b9-b108-e6f71b91d703"). InnerVolumeSpecName "kube-api-access-ckxb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:45.879258 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.879172 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-serving-cert" pod "e69fc3e3-a16c-45b9-b108-e6f71b91d703" (UID: "e69fc3e3-a16c-45b9-b108-e6f71b91d703"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:45.879359 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.879341 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-oauth-config" pod "e69fc3e3-a16c-45b9-b108-e6f71b91d703" (UID: "e69fc3e3-a16c-45b9-b108-e6f71b91d703"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:45.978014 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.977985 4335 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-serving-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:45.978014 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.978013 4335 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e69fc3e3-a16c-45b9-b108-e6f71b91d703-console-oauth-config\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:45.978160 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:45.978029 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckxb9\" (UniqueName: \"kubernetes.io/projected/e69fc3e3-a16c-45b9-b108-e6f71b91d703-kube-api-access-ckxb9\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:46.456087 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:46.456058 4335 generic.go:356] "Generic (PLEG): container finished" podID="e69fc3e3-a16c-45b9-b108-e6f71b91d703" containerID="96d50f88fff4ed30bd9f96ee549b579863f0babad0a04a215cd93a1bc5c77069" exitCode=0 Oct 30 21:45:46.456225 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:46.456112 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9875ddfb4-b4qrt" Oct 30 21:45:46.456225 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:46.456120 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9875ddfb4-b4qrt" event={"ID":"e69fc3e3-a16c-45b9-b108-e6f71b91d703","Type":"ContainerDied","Data":"96d50f88fff4ed30bd9f96ee549b579863f0babad0a04a215cd93a1bc5c77069"} Oct 30 21:45:46.456225 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:46.456162 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9875ddfb4-b4qrt" event={"ID":"e69fc3e3-a16c-45b9-b108-e6f71b91d703","Type":"ContainerDied","Data":"6fad0b82e388b829febdd311b857c903a2fe098481c2ea95a74227cfd1d1d45c"} Oct 30 21:45:46.456225 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:46.456183 4335 scope.go:122] "RemoveContainer" containerID="96d50f88fff4ed30bd9f96ee549b579863f0babad0a04a215cd93a1bc5c77069" Oct 30 21:45:46.465355 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:46.465335 4335 scope.go:122] "RemoveContainer" containerID="96d50f88fff4ed30bd9f96ee549b579863f0babad0a04a215cd93a1bc5c77069" Oct 30 21:45:46.465666 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:45:46.465646 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d50f88fff4ed30bd9f96ee549b579863f0babad0a04a215cd93a1bc5c77069\": container with ID starting with 96d50f88fff4ed30bd9f96ee549b579863f0babad0a04a215cd93a1bc5c77069 not found: ID does not exist" containerID="96d50f88fff4ed30bd9f96ee549b579863f0babad0a04a215cd93a1bc5c77069" Oct 30 21:45:46.465731 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:46.465674 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d50f88fff4ed30bd9f96ee549b579863f0babad0a04a215cd93a1bc5c77069"} err="failed to get container status \"96d50f88fff4ed30bd9f96ee549b579863f0babad0a04a215cd93a1bc5c77069\": rpc error: code = NotFound desc = could not find container \"96d50f88fff4ed30bd9f96ee549b579863f0babad0a04a215cd93a1bc5c77069\": container with ID starting with 96d50f88fff4ed30bd9f96ee549b579863f0babad0a04a215cd93a1bc5c77069 not found: ID does not exist" Oct 30 21:45:46.620246 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:46.620217 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9875ddfb4-b4qrt"] Oct 30 21:45:46.670000 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:46.669972 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9875ddfb4-b4qrt"] Oct 30 21:45:47.054630 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:45:47.054586 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e69fc3e3-a16c-45b9-b108-e6f71b91d703" path="/var/lib/kubelet/pods/e69fc3e3-a16c-45b9-b108-e6f71b91d703/volumes" Oct 30 21:45:55.321522 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:45:55.321477 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:46:04.622639 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.622551 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-d55cbfcd-q47vs" podUID="75ac4a15-4f8b-402e-a043-cd0e6b9c55e1" containerName="console" containerID="cri-o://f8de0b87092c4d94567e4633296c8b3d2249ef645c87bcd3f3b7b9c47207722d" gracePeriod=15 Oct 30 21:46:04.864849 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.864828 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:46:04.894105 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.894053 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-config\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-config\") pod \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " Oct 30 21:46:04.894105 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.894083 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-trusted-ca-bundle\") pod \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " Oct 30 21:46:04.894254 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.894111 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-oauth-serving-cert\") pod \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " Oct 30 21:46:04.894254 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.894131 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-serving-cert\") pod \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " Oct 30 21:46:04.894254 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.894165 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-tmp\" (UniqueName: \"kubernetes.io/empty-dir/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-tmp\") pod \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " Oct 30 21:46:04.894254 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.894209 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-oauth-config\") pod \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " Oct 30 21:46:04.894254 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.894240 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-service-ca\") pod \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " Oct 30 21:46:04.894500 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.894290 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-kube-api-access-kzdv2\" (UniqueName: \"kubernetes.io/projected/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-kube-api-access-kzdv2\") pod \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\" (UID: \"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1\") " Oct 30 21:46:04.894638 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.894585 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-config" pod "75ac4a15-4f8b-402e-a043-cd0e6b9c55e1" (UID: "75ac4a15-4f8b-402e-a043-cd0e6b9c55e1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:04.894638 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.894589 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-tmp" pod "75ac4a15-4f8b-402e-a043-cd0e6b9c55e1" (UID: "75ac4a15-4f8b-402e-a043-cd0e6b9c55e1"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:04.894794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.894592 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-oauth-serving-cert" pod "75ac4a15-4f8b-402e-a043-cd0e6b9c55e1" (UID: "75ac4a15-4f8b-402e-a043-cd0e6b9c55e1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:04.894794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.894725 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-trusted-ca-bundle" pod "75ac4a15-4f8b-402e-a043-cd0e6b9c55e1" (UID: "75ac4a15-4f8b-402e-a043-cd0e6b9c55e1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:04.895012 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.894901 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-service-ca" pod "75ac4a15-4f8b-402e-a043-cd0e6b9c55e1" (UID: "75ac4a15-4f8b-402e-a043-cd0e6b9c55e1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:04.896354 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.896323 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-oauth-config" pod "75ac4a15-4f8b-402e-a043-cd0e6b9c55e1" (UID: "75ac4a15-4f8b-402e-a043-cd0e6b9c55e1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:04.896494 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.896474 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-kube-api-access-kzdv2" pod "75ac4a15-4f8b-402e-a043-cd0e6b9c55e1" (UID: "75ac4a15-4f8b-402e-a043-cd0e6b9c55e1"). InnerVolumeSpecName "kube-api-access-kzdv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:04.896683 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.896665 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-serving-cert" pod "75ac4a15-4f8b-402e-a043-cd0e6b9c55e1" (UID: "75ac4a15-4f8b-402e-a043-cd0e6b9c55e1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:04.995662 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.995640 4335 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-config\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.995662 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.995662 4335 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-trusted-ca-bundle\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.995844 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.995671 4335 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-oauth-serving-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.995844 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.995679 4335 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-serving-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.995844 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.995687 4335 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-tmp\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.995844 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.995697 4335 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-console-oauth-config\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.995844 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.995705 4335 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.995844 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:04.995714 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzdv2\" (UniqueName: \"kubernetes.io/projected/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1-kube-api-access-kzdv2\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:05.521782 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:05.521748 4335 generic.go:356] "Generic (PLEG): container finished" podID="75ac4a15-4f8b-402e-a043-cd0e6b9c55e1" containerID="f8de0b87092c4d94567e4633296c8b3d2249ef645c87bcd3f3b7b9c47207722d" exitCode=0 Oct 30 21:46:05.521899 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:05.521803 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d55cbfcd-q47vs" Oct 30 21:46:05.521899 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:05.521829 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d55cbfcd-q47vs" event={"ID":"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1","Type":"ContainerDied","Data":"f8de0b87092c4d94567e4633296c8b3d2249ef645c87bcd3f3b7b9c47207722d"} Oct 30 21:46:05.521899 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:05.521871 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d55cbfcd-q47vs" event={"ID":"75ac4a15-4f8b-402e-a043-cd0e6b9c55e1","Type":"ContainerDied","Data":"1e9a31a61af8e775f351c9132b8a8d84ae4065b0a1fb744b791929b84e89259b"} Oct 30 21:46:05.521899 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:05.521892 4335 scope.go:122] "RemoveContainer" containerID="f8de0b87092c4d94567e4633296c8b3d2249ef645c87bcd3f3b7b9c47207722d" Oct 30 21:46:05.530708 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:05.530680 4335 scope.go:122] "RemoveContainer" containerID="f8de0b87092c4d94567e4633296c8b3d2249ef645c87bcd3f3b7b9c47207722d" Oct 30 21:46:05.531013 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:46:05.530994 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8de0b87092c4d94567e4633296c8b3d2249ef645c87bcd3f3b7b9c47207722d\": container with ID starting with f8de0b87092c4d94567e4633296c8b3d2249ef645c87bcd3f3b7b9c47207722d not found: ID does not exist" containerID="f8de0b87092c4d94567e4633296c8b3d2249ef645c87bcd3f3b7b9c47207722d" Oct 30 21:46:05.531077 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:05.531022 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8de0b87092c4d94567e4633296c8b3d2249ef645c87bcd3f3b7b9c47207722d"} err="failed to get container status \"f8de0b87092c4d94567e4633296c8b3d2249ef645c87bcd3f3b7b9c47207722d\": rpc error: code = NotFound desc = could not find container \"f8de0b87092c4d94567e4633296c8b3d2249ef645c87bcd3f3b7b9c47207722d\": container with ID starting with f8de0b87092c4d94567e4633296c8b3d2249ef645c87bcd3f3b7b9c47207722d not found: ID does not exist" Oct 30 21:46:05.684278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:05.684249 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d55cbfcd-q47vs"] Oct 30 21:46:05.733442 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:05.733420 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d55cbfcd-q47vs"] Oct 30 21:46:07.053986 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:46:07.053959 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ac4a15-4f8b-402e-a043-cd0e6b9c55e1" path="/var/lib/kubelet/pods/75ac4a15-4f8b-402e-a043-cd0e6b9c55e1/volumes" Oct 30 21:46:25.322308 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:46:25.322270 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:46:55.320994 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:46:55.320903 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:47:25.321494 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:47:25.321450 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:47:55.320853 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:47:55.320812 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:48:25.324677 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:48:25.322483 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:48:55.321437 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:48:55.321398 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:49:25.321561 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:49:25.321522 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:49:55.321900 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:49:55.321807 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:50:25.321688 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:50:25.321647 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:50:31.257572 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.257537 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84"] Oct 30 21:50:31.258104 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.257802 4335 cpu_manager.go:414] "containerMap: removing container" podUID="e69fc3e3-a16c-45b9-b108-e6f71b91d703" containerName="console" Oct 30 21:50:31.258104 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.257820 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e69fc3e3-a16c-45b9-b108-e6f71b91d703" containerName="console" Oct 30 21:50:31.258104 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.257834 4335 cpu_manager.go:414] "containerMap: removing container" podUID="db072ad6-3e9e-4695-9a94-c1d976d3af99" containerName="console" Oct 30 21:50:31.258104 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.257840 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db072ad6-3e9e-4695-9a94-c1d976d3af99" containerName="console" Oct 30 21:50:31.258104 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.257847 4335 cpu_manager.go:414] "containerMap: removing container" podUID="75ac4a15-4f8b-402e-a043-cd0e6b9c55e1" containerName="console" Oct 30 21:50:31.258104 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.257852 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ac4a15-4f8b-402e-a043-cd0e6b9c55e1" containerName="console" Oct 30 21:50:31.258104 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.257858 4335 cpu_manager.go:414] "containerMap: removing container" podUID="d45916e8-c498-4bf5-b5f3-df697b608ed9" containerName="insights-gathering" Oct 30 21:50:31.258104 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.257862 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d45916e8-c498-4bf5-b5f3-df697b608ed9" containerName="insights-gathering" Oct 30 21:50:31.258104 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.257911 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="e69fc3e3-a16c-45b9-b108-e6f71b91d703" containerName="console" Oct 30 21:50:31.258104 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.257919 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="db072ad6-3e9e-4695-9a94-c1d976d3af99" containerName="console" Oct 30 21:50:31.258104 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.257924 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ac4a15-4f8b-402e-a043-cd0e6b9c55e1" containerName="console" Oct 30 21:50:31.258104 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.257929 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="d45916e8-c498-4bf5-b5f3-df697b608ed9" containerName="insights-gathering" Oct 30 21:50:31.259723 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.259703 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84" Oct 30 21:50:31.323471 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.323450 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8lc5\" (UniqueName: \"kubernetes.io/projected/00124fd7-ac22-42ca-8fad-0047c9765e88-kube-api-access-c8lc5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84\" (UID: \"00124fd7-ac22-42ca-8fad-0047c9765e88\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84" Oct 30 21:50:31.323591 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.323490 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00124fd7-ac22-42ca-8fad-0047c9765e88-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84\" (UID: \"00124fd7-ac22-42ca-8fad-0047c9765e88\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84" Oct 30 21:50:31.323591 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.323525 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00124fd7-ac22-42ca-8fad-0047c9765e88-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84\" (UID: \"00124fd7-ac22-42ca-8fad-0047c9765e88\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84" Oct 30 21:50:31.360866 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.360844 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-vjb54\"" Oct 30 21:50:31.360983 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.360932 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 21:50:31.360983 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.360944 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 21:50:31.361723 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.361707 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84"] Oct 30 21:50:31.424728 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.424699 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00124fd7-ac22-42ca-8fad-0047c9765e88-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84\" (UID: \"00124fd7-ac22-42ca-8fad-0047c9765e88\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84" Oct 30 21:50:31.424868 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.424737 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8lc5\" (UniqueName: \"kubernetes.io/projected/00124fd7-ac22-42ca-8fad-0047c9765e88-kube-api-access-c8lc5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84\" (UID: \"00124fd7-ac22-42ca-8fad-0047c9765e88\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84" Oct 30 21:50:31.424933 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.424872 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00124fd7-ac22-42ca-8fad-0047c9765e88-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84\" (UID: \"00124fd7-ac22-42ca-8fad-0047c9765e88\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84" Oct 30 21:50:31.425091 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.425067 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00124fd7-ac22-42ca-8fad-0047c9765e88-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84\" (UID: \"00124fd7-ac22-42ca-8fad-0047c9765e88\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84" Oct 30 21:50:31.425136 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.425121 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00124fd7-ac22-42ca-8fad-0047c9765e88-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84\" (UID: \"00124fd7-ac22-42ca-8fad-0047c9765e88\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84" Oct 30 21:50:31.485937 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.485918 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8lc5\" (UniqueName: \"kubernetes.io/projected/00124fd7-ac22-42ca-8fad-0047c9765e88-kube-api-access-c8lc5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84\" (UID: \"00124fd7-ac22-42ca-8fad-0047c9765e88\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84" Oct 30 21:50:31.572031 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.572010 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84" Oct 30 21:50:31.940210 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.940144 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84"] Oct 30 21:50:31.944014 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:50:31.943974 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00124fd7_ac22_42ca_8fad_0047c9765e88.slice/crio-4113dbdd78d7fb011ec81925a2691bd967b0aa46565d3653eac7939718548a72 WatchSource:0}: Error finding container 4113dbdd78d7fb011ec81925a2691bd967b0aa46565d3653eac7939718548a72: Status 404 returned error can't find the container with id 4113dbdd78d7fb011ec81925a2691bd967b0aa46565d3653eac7939718548a72 Oct 30 21:50:31.946355 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:31.946335 4335 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:50:32.379149 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:32.379114 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84" event={"ID":"00124fd7-ac22-42ca-8fad-0047c9765e88","Type":"ContainerStarted","Data":"4113dbdd78d7fb011ec81925a2691bd967b0aa46565d3653eac7939718548a72"} Oct 30 21:50:32.929715 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:32.929677 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm"] Oct 30 21:50:32.931897 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:32.931878 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm" Oct 30 21:50:33.032819 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.032794 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-xq6vt\"" Oct 30 21:50:33.033853 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.033833 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm"] Oct 30 21:50:33.034553 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.034534 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2"] Oct 30 21:50:33.035350 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.035330 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c98r9\" (UniqueName: \"kubernetes.io/projected/04670690-bbaa-4329-983b-9dcb81ada2c7-kube-api-access-c98r9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm\" (UID: \"04670690-bbaa-4329-983b-9dcb81ada2c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm" Oct 30 21:50:33.035429 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.035374 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04670690-bbaa-4329-983b-9dcb81ada2c7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm\" (UID: \"04670690-bbaa-4329-983b-9dcb81ada2c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm" Oct 30 21:50:33.035514 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.035493 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04670690-bbaa-4329-983b-9dcb81ada2c7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm\" (UID: \"04670690-bbaa-4329-983b-9dcb81ada2c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm" Oct 30 21:50:33.036735 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.036718 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss"] Oct 30 21:50:33.036897 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.036877 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" Oct 30 21:50:33.038732 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.038716 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" Oct 30 21:50:33.088918 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.088883 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-ljbml\"" Oct 30 21:50:33.090230 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.090209 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-9xzfx\"" Oct 30 21:50:33.102389 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.102370 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2"] Oct 30 21:50:33.105116 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.105094 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss"] Oct 30 21:50:33.129428 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.129404 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd"] Oct 30 21:50:33.131970 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.131952 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" Oct 30 21:50:33.136150 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.136126 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04670690-bbaa-4329-983b-9dcb81ada2c7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm\" (UID: \"04670690-bbaa-4329-983b-9dcb81ada2c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm" Oct 30 21:50:33.136268 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.136175 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/140b47d4-0e7f-4e84-9d70-eaa209d01605-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2\" (UID: \"140b47d4-0e7f-4e84-9d70-eaa209d01605\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" Oct 30 21:50:33.136268 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.136207 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/140b47d4-0e7f-4e84-9d70-eaa209d01605-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2\" (UID: \"140b47d4-0e7f-4e84-9d70-eaa209d01605\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" Oct 30 21:50:33.136385 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.136323 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b117cee9-ce1b-421a-bd91-1d70a0483047-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss\" (UID: \"b117cee9-ce1b-421a-bd91-1d70a0483047\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" Oct 30 21:50:33.136385 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.136375 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shk99\" (UniqueName: \"kubernetes.io/projected/b117cee9-ce1b-421a-bd91-1d70a0483047-kube-api-access-shk99\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss\" (UID: \"b117cee9-ce1b-421a-bd91-1d70a0483047\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" Oct 30 21:50:33.136488 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.136414 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6hf7\" (UniqueName: \"kubernetes.io/projected/140b47d4-0e7f-4e84-9d70-eaa209d01605-kube-api-access-p6hf7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2\" (UID: \"140b47d4-0e7f-4e84-9d70-eaa209d01605\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" Oct 30 21:50:33.136488 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.136446 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b117cee9-ce1b-421a-bd91-1d70a0483047-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss\" (UID: \"b117cee9-ce1b-421a-bd91-1d70a0483047\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" Oct 30 21:50:33.136488 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.136479 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c98r9\" (UniqueName: \"kubernetes.io/projected/04670690-bbaa-4329-983b-9dcb81ada2c7-kube-api-access-c98r9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm\" (UID: \"04670690-bbaa-4329-983b-9dcb81ada2c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm" Oct 30 21:50:33.136658 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.136505 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04670690-bbaa-4329-983b-9dcb81ada2c7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm\" (UID: \"04670690-bbaa-4329-983b-9dcb81ada2c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm" Oct 30 21:50:33.136658 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.136524 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04670690-bbaa-4329-983b-9dcb81ada2c7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm\" (UID: \"04670690-bbaa-4329-983b-9dcb81ada2c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm" Oct 30 21:50:33.136849 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.136822 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04670690-bbaa-4329-983b-9dcb81ada2c7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm\" (UID: \"04670690-bbaa-4329-983b-9dcb81ada2c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm" Oct 30 21:50:33.189094 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.189037 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-f9kcq\"" Oct 30 21:50:33.190741 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.190716 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd"] Oct 30 21:50:33.198305 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.198283 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c98r9\" (UniqueName: \"kubernetes.io/projected/04670690-bbaa-4329-983b-9dcb81ada2c7-kube-api-access-c98r9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm\" (UID: \"04670690-bbaa-4329-983b-9dcb81ada2c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm" Oct 30 21:50:33.223286 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.223260 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6"] Oct 30 21:50:33.225581 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.225564 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" Oct 30 21:50:33.236856 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.236833 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/140b47d4-0e7f-4e84-9d70-eaa209d01605-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2\" (UID: \"140b47d4-0e7f-4e84-9d70-eaa209d01605\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" Oct 30 21:50:33.236937 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.236875 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/140b47d4-0e7f-4e84-9d70-eaa209d01605-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2\" (UID: \"140b47d4-0e7f-4e84-9d70-eaa209d01605\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" Oct 30 21:50:33.236937 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.236921 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b117cee9-ce1b-421a-bd91-1d70a0483047-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss\" (UID: \"b117cee9-ce1b-421a-bd91-1d70a0483047\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" Oct 30 21:50:33.237008 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.236953 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84db9e63-3579-46d7-a274-96e919a39275-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd\" (UID: \"84db9e63-3579-46d7-a274-96e919a39275\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" Oct 30 21:50:33.237008 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.236981 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttcg9\" (UniqueName: \"kubernetes.io/projected/84db9e63-3579-46d7-a274-96e919a39275-kube-api-access-ttcg9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd\" (UID: \"84db9e63-3579-46d7-a274-96e919a39275\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" Oct 30 21:50:33.237138 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.237011 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shk99\" (UniqueName: \"kubernetes.io/projected/b117cee9-ce1b-421a-bd91-1d70a0483047-kube-api-access-shk99\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss\" (UID: \"b117cee9-ce1b-421a-bd91-1d70a0483047\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" Oct 30 21:50:33.237138 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.237033 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84db9e63-3579-46d7-a274-96e919a39275-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd\" (UID: \"84db9e63-3579-46d7-a274-96e919a39275\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" Oct 30 21:50:33.237138 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.237060 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6hf7\" (UniqueName: \"kubernetes.io/projected/140b47d4-0e7f-4e84-9d70-eaa209d01605-kube-api-access-p6hf7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2\" (UID: \"140b47d4-0e7f-4e84-9d70-eaa209d01605\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" Oct 30 21:50:33.237138 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.237090 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b117cee9-ce1b-421a-bd91-1d70a0483047-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss\" (UID: \"b117cee9-ce1b-421a-bd91-1d70a0483047\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" Oct 30 21:50:33.237428 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.237412 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b117cee9-ce1b-421a-bd91-1d70a0483047-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss\" (UID: \"b117cee9-ce1b-421a-bd91-1d70a0483047\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" Oct 30 21:50:33.237725 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.237705 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/140b47d4-0e7f-4e84-9d70-eaa209d01605-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2\" (UID: \"140b47d4-0e7f-4e84-9d70-eaa209d01605\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" Oct 30 21:50:33.237797 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.237728 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b117cee9-ce1b-421a-bd91-1d70a0483047-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss\" (UID: \"b117cee9-ce1b-421a-bd91-1d70a0483047\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" Oct 30 21:50:33.237967 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.237948 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/140b47d4-0e7f-4e84-9d70-eaa209d01605-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2\" (UID: \"140b47d4-0e7f-4e84-9d70-eaa209d01605\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" Oct 30 21:50:33.244967 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.244944 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm" Oct 30 21:50:33.285335 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.285301 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6"] Oct 30 21:50:33.291909 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.291885 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-s5g56\"" Oct 30 21:50:33.303897 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.303815 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shk99\" (UniqueName: \"kubernetes.io/projected/b117cee9-ce1b-421a-bd91-1d70a0483047-kube-api-access-shk99\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss\" (UID: \"b117cee9-ce1b-421a-bd91-1d70a0483047\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" Oct 30 21:50:33.305706 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.305667 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6hf7\" (UniqueName: \"kubernetes.io/projected/140b47d4-0e7f-4e84-9d70-eaa209d01605-kube-api-access-p6hf7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2\" (UID: \"140b47d4-0e7f-4e84-9d70-eaa209d01605\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" Oct 30 21:50:33.337768 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.337740 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84db9e63-3579-46d7-a274-96e919a39275-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd\" (UID: \"84db9e63-3579-46d7-a274-96e919a39275\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" Oct 30 21:50:33.337896 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.337781 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttcg9\" (UniqueName: \"kubernetes.io/projected/84db9e63-3579-46d7-a274-96e919a39275-kube-api-access-ttcg9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd\" (UID: \"84db9e63-3579-46d7-a274-96e919a39275\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" Oct 30 21:50:33.337946 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.337913 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84db9e63-3579-46d7-a274-96e919a39275-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd\" (UID: \"84db9e63-3579-46d7-a274-96e919a39275\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" Oct 30 21:50:33.338000 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.337954 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e135608-c576-4f15-a0ae-6ebece176771-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6\" (UID: \"7e135608-c576-4f15-a0ae-6ebece176771\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" Oct 30 21:50:33.338053 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.338012 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj7qg\" (UniqueName: \"kubernetes.io/projected/7e135608-c576-4f15-a0ae-6ebece176771-kube-api-access-vj7qg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6\" (UID: \"7e135608-c576-4f15-a0ae-6ebece176771\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" Oct 30 21:50:33.338173 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.338151 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84db9e63-3579-46d7-a274-96e919a39275-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd\" (UID: \"84db9e63-3579-46d7-a274-96e919a39275\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" Oct 30 21:50:33.338240 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.338167 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e135608-c576-4f15-a0ae-6ebece176771-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6\" (UID: \"7e135608-c576-4f15-a0ae-6ebece176771\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" Oct 30 21:50:33.338292 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.338239 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84db9e63-3579-46d7-a274-96e919a39275-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd\" (UID: \"84db9e63-3579-46d7-a274-96e919a39275\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" Oct 30 21:50:33.352737 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.352717 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" Oct 30 21:50:33.359027 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.359009 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" Oct 30 21:50:33.399700 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.399667 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttcg9\" (UniqueName: \"kubernetes.io/projected/84db9e63-3579-46d7-a274-96e919a39275-kube-api-access-ttcg9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd\" (UID: \"84db9e63-3579-46d7-a274-96e919a39275\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" Oct 30 21:50:33.441444 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.439902 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e135608-c576-4f15-a0ae-6ebece176771-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6\" (UID: \"7e135608-c576-4f15-a0ae-6ebece176771\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" Oct 30 21:50:33.441444 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.439965 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj7qg\" (UniqueName: \"kubernetes.io/projected/7e135608-c576-4f15-a0ae-6ebece176771-kube-api-access-vj7qg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6\" (UID: \"7e135608-c576-4f15-a0ae-6ebece176771\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" Oct 30 21:50:33.441444 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.440047 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e135608-c576-4f15-a0ae-6ebece176771-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6\" (UID: \"7e135608-c576-4f15-a0ae-6ebece176771\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" Oct 30 21:50:33.441444 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.440446 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e135608-c576-4f15-a0ae-6ebece176771-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6\" (UID: \"7e135608-c576-4f15-a0ae-6ebece176771\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" Oct 30 21:50:33.441892 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.441850 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e135608-c576-4f15-a0ae-6ebece176771-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6\" (UID: \"7e135608-c576-4f15-a0ae-6ebece176771\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" Oct 30 21:50:33.447973 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.447719 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" Oct 30 21:50:33.502560 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.502473 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj7qg\" (UniqueName: \"kubernetes.io/projected/7e135608-c576-4f15-a0ae-6ebece176771-kube-api-access-vj7qg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6\" (UID: \"7e135608-c576-4f15-a0ae-6ebece176771\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" Oct 30 21:50:33.539845 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.539812 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" Oct 30 21:50:33.654445 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.654412 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm"] Oct 30 21:50:33.657230 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:50:33.657202 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04670690_bbaa_4329_983b_9dcb81ada2c7.slice/crio-4bda04f679d03f52e8c8516ea5d185e7eeda6dc5733027a491310c240946fcc7 WatchSource:0}: Error finding container 4bda04f679d03f52e8c8516ea5d185e7eeda6dc5733027a491310c240946fcc7: Status 404 returned error can't find the container with id 4bda04f679d03f52e8c8516ea5d185e7eeda6dc5733027a491310c240946fcc7 Oct 30 21:50:33.735446 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.735318 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2"] Oct 30 21:50:33.737968 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:50:33.737940 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140b47d4_0e7f_4e84_9d70_eaa209d01605.slice/crio-3e998bc1d2a9c179398d9d41bd442f9d03dd8e202d071c926888dcfc704384cd WatchSource:0}: Error finding container 3e998bc1d2a9c179398d9d41bd442f9d03dd8e202d071c926888dcfc704384cd: Status 404 returned error can't find the container with id 3e998bc1d2a9c179398d9d41bd442f9d03dd8e202d071c926888dcfc704384cd Oct 30 21:50:33.766840 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.766820 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss"] Oct 30 21:50:33.769434 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:50:33.769412 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb117cee9_ce1b_421a_bd91_1d70a0483047.slice/crio-57802345b2d8b073a9193158cb049520111232605a6b65c737fe141e28aa4084 WatchSource:0}: Error finding container 57802345b2d8b073a9193158cb049520111232605a6b65c737fe141e28aa4084: Status 404 returned error can't find the container with id 57802345b2d8b073a9193158cb049520111232605a6b65c737fe141e28aa4084 Oct 30 21:50:33.837916 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.837895 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd"] Oct 30 21:50:33.840254 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:50:33.840226 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84db9e63_3579_46d7_a274_96e919a39275.slice/crio-54bc53c4585a4e277d759ff163d79932deed6bbe5ccb77453cac672bc547c801 WatchSource:0}: Error finding container 54bc53c4585a4e277d759ff163d79932deed6bbe5ccb77453cac672bc547c801: Status 404 returned error can't find the container with id 54bc53c4585a4e277d759ff163d79932deed6bbe5ccb77453cac672bc547c801 Oct 30 21:50:33.916594 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:33.916554 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6"] Oct 30 21:50:33.919708 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:50:33.919685 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e135608_c576_4f15_a0ae_6ebece176771.slice/crio-f1bc0e2a29215d42ea03742db7c0460f96e43e5b88e67375f387257422e4d648 WatchSource:0}: Error finding container f1bc0e2a29215d42ea03742db7c0460f96e43e5b88e67375f387257422e4d648: Status 404 returned error can't find the container with id f1bc0e2a29215d42ea03742db7c0460f96e43e5b88e67375f387257422e4d648 Oct 30 21:50:34.389074 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:34.389034 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" event={"ID":"84db9e63-3579-46d7-a274-96e919a39275","Type":"ContainerStarted","Data":"54bc53c4585a4e277d759ff163d79932deed6bbe5ccb77453cac672bc547c801"} Oct 30 21:50:34.390430 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:34.390397 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" event={"ID":"7e135608-c576-4f15-a0ae-6ebece176771","Type":"ContainerStarted","Data":"f1bc0e2a29215d42ea03742db7c0460f96e43e5b88e67375f387257422e4d648"} Oct 30 21:50:34.391746 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:34.391716 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm" event={"ID":"04670690-bbaa-4329-983b-9dcb81ada2c7","Type":"ContainerStarted","Data":"4bda04f679d03f52e8c8516ea5d185e7eeda6dc5733027a491310c240946fcc7"} Oct 30 21:50:34.393072 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:34.393049 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" event={"ID":"140b47d4-0e7f-4e84-9d70-eaa209d01605","Type":"ContainerStarted","Data":"3e998bc1d2a9c179398d9d41bd442f9d03dd8e202d071c926888dcfc704384cd"} Oct 30 21:50:34.394368 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:34.394347 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" event={"ID":"b117cee9-ce1b-421a-bd91-1d70a0483047","Type":"ContainerStarted","Data":"57802345b2d8b073a9193158cb049520111232605a6b65c737fe141e28aa4084"} Oct 30 21:50:34.931123 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:34.931086 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4"] Oct 30 21:50:34.934105 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:34.934083 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4" Oct 30 21:50:35.035319 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:35.035283 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-jm2c7\"" Oct 30 21:50:35.036800 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:35.036776 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4"] Oct 30 21:50:35.052425 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:35.052072 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkqdq\" (UniqueName: \"kubernetes.io/projected/cd8714dd-e64c-440b-b110-6a2d087e5233-kube-api-access-jkqdq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4\" (UID: \"cd8714dd-e64c-440b-b110-6a2d087e5233\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4" Oct 30 21:50:35.052425 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:35.052118 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd8714dd-e64c-440b-b110-6a2d087e5233-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4\" (UID: \"cd8714dd-e64c-440b-b110-6a2d087e5233\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4" Oct 30 21:50:35.052425 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:35.052191 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd8714dd-e64c-440b-b110-6a2d087e5233-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4\" (UID: \"cd8714dd-e64c-440b-b110-6a2d087e5233\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4" Oct 30 21:50:35.153603 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:35.153568 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd8714dd-e64c-440b-b110-6a2d087e5233-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4\" (UID: \"cd8714dd-e64c-440b-b110-6a2d087e5233\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4" Oct 30 21:50:35.153783 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:35.153664 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkqdq\" (UniqueName: \"kubernetes.io/projected/cd8714dd-e64c-440b-b110-6a2d087e5233-kube-api-access-jkqdq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4\" (UID: \"cd8714dd-e64c-440b-b110-6a2d087e5233\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4" Oct 30 21:50:35.153783 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:35.153714 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd8714dd-e64c-440b-b110-6a2d087e5233-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4\" (UID: \"cd8714dd-e64c-440b-b110-6a2d087e5233\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4" Oct 30 21:50:35.154948 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:35.154315 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd8714dd-e64c-440b-b110-6a2d087e5233-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4\" (UID: \"cd8714dd-e64c-440b-b110-6a2d087e5233\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4" Oct 30 21:50:35.154948 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:35.154629 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd8714dd-e64c-440b-b110-6a2d087e5233-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4\" (UID: \"cd8714dd-e64c-440b-b110-6a2d087e5233\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4" Oct 30 21:50:35.227773 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:35.227644 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkqdq\" (UniqueName: \"kubernetes.io/projected/cd8714dd-e64c-440b-b110-6a2d087e5233-kube-api-access-jkqdq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4\" (UID: \"cd8714dd-e64c-440b-b110-6a2d087e5233\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4" Oct 30 21:50:35.251078 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:35.250908 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4" Oct 30 21:50:35.686633 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:35.679787 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4"] Oct 30 21:50:35.690792 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:50:35.690769 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8714dd_e64c_440b_b110_6a2d087e5233.slice/crio-c105a8ed4a3f33fb2c9dfd58b44c3cc4b0d46d7c924f17d6f8cb1c52f81797e5 WatchSource:0}: Error finding container c105a8ed4a3f33fb2c9dfd58b44c3cc4b0d46d7c924f17d6f8cb1c52f81797e5: Status 404 returned error can't find the container with id c105a8ed4a3f33fb2c9dfd58b44c3cc4b0d46d7c924f17d6f8cb1c52f81797e5 Oct 30 21:50:36.409234 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:36.409189 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4" event={"ID":"cd8714dd-e64c-440b-b110-6a2d087e5233","Type":"ContainerStarted","Data":"c105a8ed4a3f33fb2c9dfd58b44c3cc4b0d46d7c924f17d6f8cb1c52f81797e5"} Oct 30 21:50:37.530492 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:37.530461 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66"] Oct 30 21:50:37.536208 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:37.536174 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" Oct 30 21:50:37.638929 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:37.638902 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66"] Oct 30 21:50:37.676213 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:37.676184 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cca7fe9d-c55b-4597-843d-a9cf78efc242-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66\" (UID: \"cca7fe9d-c55b-4597-843d-a9cf78efc242\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" Oct 30 21:50:37.676380 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:37.676242 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk6cb\" (UniqueName: \"kubernetes.io/projected/cca7fe9d-c55b-4597-843d-a9cf78efc242-kube-api-access-qk6cb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66\" (UID: \"cca7fe9d-c55b-4597-843d-a9cf78efc242\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" Oct 30 21:50:37.676380 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:37.676304 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cca7fe9d-c55b-4597-843d-a9cf78efc242-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66\" (UID: \"cca7fe9d-c55b-4597-843d-a9cf78efc242\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" Oct 30 21:50:37.777391 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:37.777352 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cca7fe9d-c55b-4597-843d-a9cf78efc242-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66\" (UID: \"cca7fe9d-c55b-4597-843d-a9cf78efc242\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" Oct 30 21:50:37.777570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:37.777420 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk6cb\" (UniqueName: \"kubernetes.io/projected/cca7fe9d-c55b-4597-843d-a9cf78efc242-kube-api-access-qk6cb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66\" (UID: \"cca7fe9d-c55b-4597-843d-a9cf78efc242\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" Oct 30 21:50:37.777570 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:37.777470 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cca7fe9d-c55b-4597-843d-a9cf78efc242-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66\" (UID: \"cca7fe9d-c55b-4597-843d-a9cf78efc242\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" Oct 30 21:50:37.778035 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:37.777917 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cca7fe9d-c55b-4597-843d-a9cf78efc242-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66\" (UID: \"cca7fe9d-c55b-4597-843d-a9cf78efc242\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" Oct 30 21:50:37.778035 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:37.777928 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cca7fe9d-c55b-4597-843d-a9cf78efc242-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66\" (UID: \"cca7fe9d-c55b-4597-843d-a9cf78efc242\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" Oct 30 21:50:37.841601 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:37.841572 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk6cb\" (UniqueName: \"kubernetes.io/projected/cca7fe9d-c55b-4597-843d-a9cf78efc242-kube-api-access-qk6cb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66\" (UID: \"cca7fe9d-c55b-4597-843d-a9cf78efc242\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" Oct 30 21:50:37.858529 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:37.858365 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" Oct 30 21:50:38.377582 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:38.377406 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66"] Oct 30 21:50:38.385299 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:50:38.385237 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca7fe9d_c55b_4597_843d_a9cf78efc242.slice/crio-ba70986d22790071803a7858e190b0c897caf5998aab21f41fce27ed9d8c1b50 WatchSource:0}: Error finding container ba70986d22790071803a7858e190b0c897caf5998aab21f41fce27ed9d8c1b50: Status 404 returned error can't find the container with id ba70986d22790071803a7858e190b0c897caf5998aab21f41fce27ed9d8c1b50 Oct 30 21:50:38.423497 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:38.423462 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" event={"ID":"cca7fe9d-c55b-4597-843d-a9cf78efc242","Type":"ContainerStarted","Data":"ba70986d22790071803a7858e190b0c897caf5998aab21f41fce27ed9d8c1b50"} Oct 30 21:50:38.948928 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:38.948893 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg"] Oct 30 21:50:38.952436 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:38.952411 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.006599 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.006576 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v5mg8\"" Oct 30 21:50:39.006750 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.006687 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:50:39.006809 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.006752 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:50:39.006873 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.006821 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:50:39.015514 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.014796 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg"] Oct 30 21:50:39.102595 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.102561 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.102814 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.102653 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.102814 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.102684 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fba568-9213-4aa9-a093-17df3f0c506e-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.102814 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.102711 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.102814 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.102752 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nmq5\" (UniqueName: \"kubernetes.io/projected/91fba568-9213-4aa9-a093-17df3f0c506e-kube-api-access-6nmq5\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.102814 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.102780 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fba568-9213-4aa9-a093-17df3f0c506e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.102814 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.102812 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.103187 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.102849 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fba568-9213-4aa9-a093-17df3f0c506e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.103187 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.102919 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.204267 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.203474 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.204267 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.203517 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fba568-9213-4aa9-a093-17df3f0c506e-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.204267 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.203576 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.204267 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.203655 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nmq5\" (UniqueName: \"kubernetes.io/projected/91fba568-9213-4aa9-a093-17df3f0c506e-kube-api-access-6nmq5\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.204267 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.203685 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fba568-9213-4aa9-a093-17df3f0c506e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.204267 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.203716 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.204267 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.204160 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.204267 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.203750 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fba568-9213-4aa9-a093-17df3f0c506e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.205174 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.204301 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.205174 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.204333 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.205174 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.204359 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.205174 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.204819 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.205487 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.205346 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fba568-9213-4aa9-a093-17df3f0c506e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.205777 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.205754 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.207794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.207722 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fba568-9213-4aa9-a093-17df3f0c506e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.219800 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.219772 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.266729 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.266700 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w"] Oct 30 21:50:39.278051 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.277989 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nmq5\" (UniqueName: \"kubernetes.io/projected/91fba568-9213-4aa9-a093-17df3f0c506e-kube-api-access-6nmq5\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.278987 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.278963 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fba568-9213-4aa9-a093-17df3f0c506e-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-hbnxg\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.279602 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.279555 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.332207 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.332158 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w"] Oct 30 21:50:39.333650 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.333496 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-q7hdz\"" Oct 30 21:50:39.407819 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.407291 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec4cf36-264a-40d2-9a17-1d9e712d1238-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.407819 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.407357 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec4cf36-264a-40d2-9a17-1d9e712d1238-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.407819 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.407395 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec4cf36-264a-40d2-9a17-1d9e712d1238-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.407819 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.407420 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec4cf36-264a-40d2-9a17-1d9e712d1238-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.407819 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.407462 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec4cf36-264a-40d2-9a17-1d9e712d1238-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.407819 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.407543 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec4cf36-264a-40d2-9a17-1d9e712d1238-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.407819 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.407572 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnx29\" (UniqueName: \"kubernetes.io/projected/dec4cf36-264a-40d2-9a17-1d9e712d1238-kube-api-access-jnx29\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.407819 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.407601 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec4cf36-264a-40d2-9a17-1d9e712d1238-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.407819 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.407644 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec4cf36-264a-40d2-9a17-1d9e712d1238-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.509072 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.509004 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec4cf36-264a-40d2-9a17-1d9e712d1238-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.509072 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.509053 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnx29\" (UniqueName: \"kubernetes.io/projected/dec4cf36-264a-40d2-9a17-1d9e712d1238-kube-api-access-jnx29\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.509288 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.509085 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec4cf36-264a-40d2-9a17-1d9e712d1238-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.509288 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.509115 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec4cf36-264a-40d2-9a17-1d9e712d1238-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.509288 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.509154 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec4cf36-264a-40d2-9a17-1d9e712d1238-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.509288 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.509185 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec4cf36-264a-40d2-9a17-1d9e712d1238-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.509288 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.509227 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec4cf36-264a-40d2-9a17-1d9e712d1238-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.509288 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.509251 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec4cf36-264a-40d2-9a17-1d9e712d1238-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.509590 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.509294 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec4cf36-264a-40d2-9a17-1d9e712d1238-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.510584 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.510560 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec4cf36-264a-40d2-9a17-1d9e712d1238-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.511386 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.510832 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec4cf36-264a-40d2-9a17-1d9e712d1238-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.511386 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.510823 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec4cf36-264a-40d2-9a17-1d9e712d1238-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.511386 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.511081 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec4cf36-264a-40d2-9a17-1d9e712d1238-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.511386 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.511235 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec4cf36-264a-40d2-9a17-1d9e712d1238-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.514381 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.514336 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec4cf36-264a-40d2-9a17-1d9e712d1238-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.515700 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.515656 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec4cf36-264a-40d2-9a17-1d9e712d1238-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.577515 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.577470 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:39.578226 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.578192 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec4cf36-264a-40d2-9a17-1d9e712d1238-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.582125 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.582097 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnx29\" (UniqueName: \"kubernetes.io/projected/dec4cf36-264a-40d2-9a17-1d9e712d1238-kube-api-access-jnx29\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w\" (UID: \"dec4cf36-264a-40d2-9a17-1d9e712d1238\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:39.614238 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:39.613914 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:40.110714 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.110688 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg"] Oct 30 21:50:40.121779 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:50:40.121741 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fba568_9213_4aa9_a093_17df3f0c506e.slice/crio-463f2b0f674a53530989a1d7e44aac1476ad0a76f551b21a63e8a2cf552a5cdc WatchSource:0}: Error finding container 463f2b0f674a53530989a1d7e44aac1476ad0a76f551b21a63e8a2cf552a5cdc: Status 404 returned error can't find the container with id 463f2b0f674a53530989a1d7e44aac1476ad0a76f551b21a63e8a2cf552a5cdc Oct 30 21:50:40.161449 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.160496 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w"] Oct 30 21:50:40.163966 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:50:40.163926 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec4cf36_264a_40d2_9a17_1d9e712d1238.slice/crio-b2124d8d5b2cde944888707ba8adccd82efa18aa0128e97b9769c7aa2fd383d2 WatchSource:0}: Error finding container b2124d8d5b2cde944888707ba8adccd82efa18aa0128e97b9769c7aa2fd383d2: Status 404 returned error can't find the container with id b2124d8d5b2cde944888707ba8adccd82efa18aa0128e97b9769c7aa2fd383d2 Oct 30 21:50:40.459735 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.459657 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" event={"ID":"91fba568-9213-4aa9-a093-17df3f0c506e","Type":"ContainerStarted","Data":"463f2b0f674a53530989a1d7e44aac1476ad0a76f551b21a63e8a2cf552a5cdc"} Oct 30 21:50:40.467071 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.467041 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" event={"ID":"dec4cf36-264a-40d2-9a17-1d9e712d1238","Type":"ContainerStarted","Data":"b2124d8d5b2cde944888707ba8adccd82efa18aa0128e97b9769c7aa2fd383d2"} Oct 30 21:50:40.585910 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.585878 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx"] Oct 30 21:50:40.590722 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.590697 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" Oct 30 21:50:40.618322 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.617043 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp"] Oct 30 21:50:40.620902 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.620521 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" Oct 30 21:50:40.648279 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.647924 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7"] Oct 30 21:50:40.655568 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.655081 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx"] Oct 30 21:50:40.655568 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.655197 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" Oct 30 21:50:40.679932 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.678137 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd"] Oct 30 21:50:40.682037 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.682017 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd" Oct 30 21:50:40.688009 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.687968 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp"] Oct 30 21:50:40.710066 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.709928 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7"] Oct 30 21:50:40.721822 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.721797 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a6e38bc-3e75-4392-90b2-a0472b37a443-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx\" (UID: \"2a6e38bc-3e75-4392-90b2-a0472b37a443\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" Oct 30 21:50:40.721929 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.721859 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp\" (UID: \"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" Oct 30 21:50:40.721929 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.721893 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvwp\" (UniqueName: \"kubernetes.io/projected/2a6e38bc-3e75-4392-90b2-a0472b37a443-kube-api-access-vfvwp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx\" (UID: \"2a6e38bc-3e75-4392-90b2-a0472b37a443\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" Oct 30 21:50:40.721929 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.721922 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp\" (UID: \"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" Oct 30 21:50:40.722099 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.721972 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a6e38bc-3e75-4392-90b2-a0472b37a443-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx\" (UID: \"2a6e38bc-3e75-4392-90b2-a0472b37a443\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" Oct 30 21:50:40.722099 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.722014 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gpxv\" (UniqueName: \"kubernetes.io/projected/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-kube-api-access-5gpxv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp\" (UID: \"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" Oct 30 21:50:40.774776 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.774706 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd"] Oct 30 21:50:40.822867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.822809 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a6e38bc-3e75-4392-90b2-a0472b37a443-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx\" (UID: \"2a6e38bc-3e75-4392-90b2-a0472b37a443\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" Oct 30 21:50:40.822983 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.822927 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7\" (UID: \"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" Oct 30 21:50:40.822983 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.822965 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ht57\" (UniqueName: \"kubernetes.io/projected/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-kube-api-access-4ht57\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7\" (UID: \"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" Oct 30 21:50:40.823067 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.823000 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7\" (UID: \"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" Oct 30 21:50:40.823067 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.823028 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp\" (UID: \"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" Oct 30 21:50:40.823067 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.823054 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b128ca38-8f49-4584-a0d0-f87fdc422db6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd\" (UID: \"b128ca38-8f49-4584-a0d0-f87fdc422db6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd" Oct 30 21:50:40.823155 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.823083 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvwp\" (UniqueName: \"kubernetes.io/projected/2a6e38bc-3e75-4392-90b2-a0472b37a443-kube-api-access-vfvwp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx\" (UID: \"2a6e38bc-3e75-4392-90b2-a0472b37a443\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" Oct 30 21:50:40.823155 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.823113 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp\" (UID: \"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" Oct 30 21:50:40.823155 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.823140 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x2rx\" (UniqueName: \"kubernetes.io/projected/b128ca38-8f49-4584-a0d0-f87fdc422db6-kube-api-access-6x2rx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd\" (UID: \"b128ca38-8f49-4584-a0d0-f87fdc422db6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd" Oct 30 21:50:40.823236 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.823222 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a6e38bc-3e75-4392-90b2-a0472b37a443-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx\" (UID: \"2a6e38bc-3e75-4392-90b2-a0472b37a443\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" Oct 30 21:50:40.823277 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.823247 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b128ca38-8f49-4584-a0d0-f87fdc422db6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd\" (UID: \"b128ca38-8f49-4584-a0d0-f87fdc422db6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd" Oct 30 21:50:40.823341 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.823317 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gpxv\" (UniqueName: \"kubernetes.io/projected/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-kube-api-access-5gpxv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp\" (UID: \"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" Oct 30 21:50:40.824045 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.824013 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a6e38bc-3e75-4392-90b2-a0472b37a443-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx\" (UID: \"2a6e38bc-3e75-4392-90b2-a0472b37a443\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" Oct 30 21:50:40.824467 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.824445 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp\" (UID: \"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" Oct 30 21:50:40.824982 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.824960 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp\" (UID: \"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" Oct 30 21:50:40.825225 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.825207 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a6e38bc-3e75-4392-90b2-a0472b37a443-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx\" (UID: \"2a6e38bc-3e75-4392-90b2-a0472b37a443\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" Oct 30 21:50:40.905556 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.905521 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gpxv\" (UniqueName: \"kubernetes.io/projected/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-kube-api-access-5gpxv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp\" (UID: \"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" Oct 30 21:50:40.910281 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.910224 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvwp\" (UniqueName: \"kubernetes.io/projected/2a6e38bc-3e75-4392-90b2-a0472b37a443-kube-api-access-vfvwp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx\" (UID: \"2a6e38bc-3e75-4392-90b2-a0472b37a443\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" Oct 30 21:50:40.913040 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.913017 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" Oct 30 21:50:40.924586 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.924563 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7\" (UID: \"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" Oct 30 21:50:40.924756 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.924607 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ht57\" (UniqueName: \"kubernetes.io/projected/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-kube-api-access-4ht57\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7\" (UID: \"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" Oct 30 21:50:40.924756 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.924657 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7\" (UID: \"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" Oct 30 21:50:40.924756 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.924693 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b128ca38-8f49-4584-a0d0-f87fdc422db6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd\" (UID: \"b128ca38-8f49-4584-a0d0-f87fdc422db6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd" Oct 30 21:50:40.924756 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.924732 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x2rx\" (UniqueName: \"kubernetes.io/projected/b128ca38-8f49-4584-a0d0-f87fdc422db6-kube-api-access-6x2rx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd\" (UID: \"b128ca38-8f49-4584-a0d0-f87fdc422db6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd" Oct 30 21:50:40.925048 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.924803 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b128ca38-8f49-4584-a0d0-f87fdc422db6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd\" (UID: \"b128ca38-8f49-4584-a0d0-f87fdc422db6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd" Oct 30 21:50:40.925278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.925254 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b128ca38-8f49-4584-a0d0-f87fdc422db6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd\" (UID: \"b128ca38-8f49-4584-a0d0-f87fdc422db6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd" Oct 30 21:50:40.925588 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.925568 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7\" (UID: \"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" Oct 30 21:50:40.926127 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.926089 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7\" (UID: \"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" Oct 30 21:50:40.926345 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.926328 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b128ca38-8f49-4584-a0d0-f87fdc422db6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd\" (UID: \"b128ca38-8f49-4584-a0d0-f87fdc422db6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd" Oct 30 21:50:40.939896 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.939876 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" Oct 30 21:50:40.994714 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.994683 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ht57\" (UniqueName: \"kubernetes.io/projected/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-kube-api-access-4ht57\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7\" (UID: \"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" Oct 30 21:50:40.995608 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:40.995580 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x2rx\" (UniqueName: \"kubernetes.io/projected/b128ca38-8f49-4584-a0d0-f87fdc422db6-kube-api-access-6x2rx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd\" (UID: \"b128ca38-8f49-4584-a0d0-f87fdc422db6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd" Oct 30 21:50:41.003335 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:41.003304 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd" Oct 30 21:50:41.275211 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:41.274938 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" Oct 30 21:50:41.379687 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:41.379452 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp"] Oct 30 21:50:41.435369 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:50:41.435320 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6e38bc_3e75_4392_90b2_a0472b37a443.slice/crio-d7cf203381ff71c20947fbc460c579b55439ccdb7c77189506eafa565d22d304 WatchSource:0}: Error finding container d7cf203381ff71c20947fbc460c579b55439ccdb7c77189506eafa565d22d304: Status 404 returned error can't find the container with id d7cf203381ff71c20947fbc460c579b55439ccdb7c77189506eafa565d22d304 Oct 30 21:50:41.435485 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:41.435395 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx"] Oct 30 21:50:41.476587 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:41.476552 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" event={"ID":"2a6e38bc-3e75-4392-90b2-a0472b37a443","Type":"ContainerStarted","Data":"d7cf203381ff71c20947fbc460c579b55439ccdb7c77189506eafa565d22d304"} Oct 30 21:50:41.478093 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:41.478063 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" event={"ID":"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7","Type":"ContainerStarted","Data":"60973549d9dc79956f724a48a55615a0e04efb7ebe1789b2219010285bf5c790"} Oct 30 21:50:41.484069 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:41.484039 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd"] Oct 30 21:50:41.494927 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:50:41.494890 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb128ca38_8f49_4584_a0d0_f87fdc422db6.slice/crio-7570d1a81cb3d33f55a46a03c85d5908b50c570d2ea8f66aaf871fbafa2e268b WatchSource:0}: Error finding container 7570d1a81cb3d33f55a46a03c85d5908b50c570d2ea8f66aaf871fbafa2e268b: Status 404 returned error can't find the container with id 7570d1a81cb3d33f55a46a03c85d5908b50c570d2ea8f66aaf871fbafa2e268b Oct 30 21:50:41.728710 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:41.727994 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7"] Oct 30 21:50:41.735369 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:50:41.735269 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7a1ea3_fd94_4dfb_85c0_66d2a193fd03.slice/crio-d286b414b2ed97e34e612f807039597fd2a5ebc6cca445e2474bb669eebb0a21 WatchSource:0}: Error finding container d286b414b2ed97e34e612f807039597fd2a5ebc6cca445e2474bb669eebb0a21: Status 404 returned error can't find the container with id d286b414b2ed97e34e612f807039597fd2a5ebc6cca445e2474bb669eebb0a21 Oct 30 21:50:42.500169 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:42.500083 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd" event={"ID":"b128ca38-8f49-4584-a0d0-f87fdc422db6","Type":"ContainerStarted","Data":"7570d1a81cb3d33f55a46a03c85d5908b50c570d2ea8f66aaf871fbafa2e268b"} Oct 30 21:50:42.533166 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:42.533099 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" event={"ID":"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03","Type":"ContainerStarted","Data":"d286b414b2ed97e34e612f807039597fd2a5ebc6cca445e2474bb669eebb0a21"} Oct 30 21:50:50.638689 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:50.637414 4335 generic.go:356] "Generic (PLEG): container finished" podID="00124fd7-ac22-42ca-8fad-0047c9765e88" containerID="9d9c6bbfd88769dc7b60ec8258463610483f7f350745d49765daceffad1f887a" exitCode=0 Oct 30 21:50:50.638689 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:50.637547 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84" event={"ID":"00124fd7-ac22-42ca-8fad-0047c9765e88","Type":"ContainerDied","Data":"9d9c6bbfd88769dc7b60ec8258463610483f7f350745d49765daceffad1f887a"} Oct 30 21:50:53.661650 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:53.661498 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84" event={"ID":"00124fd7-ac22-42ca-8fad-0047c9765e88","Type":"ContainerDied","Data":"8eff32e20af7293eefde3d74fd5c10ef12cff257bce0b3ce0cd96315fcdd72d1"} Oct 30 21:50:53.661650 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:53.660601 4335 generic.go:356] "Generic (PLEG): container finished" podID="00124fd7-ac22-42ca-8fad-0047c9765e88" containerID="8eff32e20af7293eefde3d74fd5c10ef12cff257bce0b3ce0cd96315fcdd72d1" exitCode=0 Oct 30 21:50:53.684847 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:53.684809 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:50:53.684943 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:53.684887 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:50:53.684943 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:53.684933 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:50:53.703819 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:53.702714 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:50:53.703819 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:53.702784 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:50:53.703819 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:53.702828 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:50:54.689734 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:54.689690 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" event={"ID":"dec4cf36-264a-40d2-9a17-1d9e712d1238","Type":"ContainerStarted","Data":"c72f598dfbbb9dee6136140c7d2e41079f1ad7ef02cf1b150a20b63035b40920"} Oct 30 21:50:54.705208 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:54.703451 4335 generic.go:356] "Generic (PLEG): container finished" podID="cca7fe9d-c55b-4597-843d-a9cf78efc242" containerID="08c98d8b74ca8531db264281e747ee553766170b85e12a14ea9264ccc3abc273" exitCode=0 Oct 30 21:50:54.705208 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:54.703549 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" event={"ID":"cca7fe9d-c55b-4597-843d-a9cf78efc242","Type":"ContainerDied","Data":"08c98d8b74ca8531db264281e747ee553766170b85e12a14ea9264ccc3abc273"} Oct 30 21:50:54.713336 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:54.712816 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" event={"ID":"91fba568-9213-4aa9-a093-17df3f0c506e","Type":"ContainerStarted","Data":"f091890b79771482e31d2f6a353e6e698c91a23f792aae79e8ffcce605ab2441"} Oct 30 21:50:54.814043 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:54.813691 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" podStartSLOduration=2.278457795 podStartE2EDuration="15.813670415s" totalImagesPullingTime="13.53521262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:54.812497288 +0000 UTC m=+387.912929024" watchObservedRunningTime="2026-10-30 21:50:54.813670415 +0000 UTC m=+387.914102130" Oct 30 21:50:55.052370 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:55.051994 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" podStartSLOduration=3.50028053 podStartE2EDuration="17.051975844s" totalImagesPullingTime="13.551695314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:55.049701141 +0000 UTC m=+388.150132874" watchObservedRunningTime="2026-10-30 21:50:55.051975844 +0000 UTC m=+388.152407579" Oct 30 21:50:55.342310 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:50:55.342231 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:50:55.578460 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:55.578428 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:55.585179 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:55.584916 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:55.616067 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:55.614393 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:55.620576 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:55.620462 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:55.720939 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:55.720725 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:55.720939 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:55.720767 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:55.721904 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:55.721887 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:50:55.722019 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:55.721952 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w" Oct 30 21:50:56.730737 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:56.730639 4335 generic.go:356] "Generic (PLEG): container finished" podID="cca7fe9d-c55b-4597-843d-a9cf78efc242" containerID="c66df27020dc208dbda11a10828df6d155c7b813d3bdecc88e64396b405b56c9" exitCode=0 Oct 30 21:50:56.731116 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:56.730746 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" event={"ID":"cca7fe9d-c55b-4597-843d-a9cf78efc242","Type":"ContainerDied","Data":"c66df27020dc208dbda11a10828df6d155c7b813d3bdecc88e64396b405b56c9"} Oct 30 21:50:56.735677 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:56.735171 4335 generic.go:356] "Generic (PLEG): container finished" podID="84db9e63-3579-46d7-a274-96e919a39275" containerID="4d733f86d837cb843c4ec41150cc91c4f4beaf3fc74f7e0b1a705680eb420159" exitCode=0 Oct 30 21:50:56.735677 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:56.735257 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" event={"ID":"84db9e63-3579-46d7-a274-96e919a39275","Type":"ContainerDied","Data":"4d733f86d837cb843c4ec41150cc91c4f4beaf3fc74f7e0b1a705680eb420159"} Oct 30 21:50:56.737565 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:56.737541 4335 generic.go:356] "Generic (PLEG): container finished" podID="7e135608-c576-4f15-a0ae-6ebece176771" containerID="21f2362e3c6e3588b744ea39d5a7e96b150194588a1471dd9fc2de4c600aa6ce" exitCode=0 Oct 30 21:50:56.737667 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:56.737625 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" event={"ID":"7e135608-c576-4f15-a0ae-6ebece176771","Type":"ContainerDied","Data":"21f2362e3c6e3588b744ea39d5a7e96b150194588a1471dd9fc2de4c600aa6ce"} Oct 30 21:50:56.740758 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:56.739842 4335 generic.go:356] "Generic (PLEG): container finished" podID="04670690-bbaa-4329-983b-9dcb81ada2c7" containerID="f6876cf78f9a130b794967073ae2d8613615bd2abb6ebfc4d0efbffe18b3b7d7" exitCode=0 Oct 30 21:50:56.740758 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:56.739923 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm" event={"ID":"04670690-bbaa-4329-983b-9dcb81ada2c7","Type":"ContainerDied","Data":"f6876cf78f9a130b794967073ae2d8613615bd2abb6ebfc4d0efbffe18b3b7d7"} Oct 30 21:50:56.748129 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:56.748107 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" event={"ID":"140b47d4-0e7f-4e84-9d70-eaa209d01605","Type":"ContainerStarted","Data":"28ee00638a834732667dfcf48ab2b8a845f967b389d9398d46ff068909ccbe3a"} Oct 30 21:50:56.751942 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:56.751919 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" event={"ID":"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03","Type":"ContainerStarted","Data":"40ce500be9f3187e2ba92df7e74a72500a4159b6f14a949cf9f66610699053b9"} Oct 30 21:50:57.758430 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:57.758392 4335 generic.go:356] "Generic (PLEG): container finished" podID="140b47d4-0e7f-4e84-9d70-eaa209d01605" containerID="28ee00638a834732667dfcf48ab2b8a845f967b389d9398d46ff068909ccbe3a" exitCode=0 Oct 30 21:50:57.758430 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:57.758426 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" event={"ID":"140b47d4-0e7f-4e84-9d70-eaa209d01605","Type":"ContainerDied","Data":"28ee00638a834732667dfcf48ab2b8a845f967b389d9398d46ff068909ccbe3a"} Oct 30 21:50:57.760299 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:57.760274 4335 generic.go:356] "Generic (PLEG): container finished" podID="f4ca5459-710c-48ee-9907-b9f1c0fb5fd7" containerID="879d90fed766acaaf66098bb748441b30eaff03a8ade202ba8c9875d69860560" exitCode=0 Oct 30 21:50:57.760429 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:57.760371 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" event={"ID":"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7","Type":"ContainerDied","Data":"879d90fed766acaaf66098bb748441b30eaff03a8ade202ba8c9875d69860560"} Oct 30 21:50:57.762322 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:57.762297 4335 generic.go:356] "Generic (PLEG): container finished" podID="0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03" containerID="40ce500be9f3187e2ba92df7e74a72500a4159b6f14a949cf9f66610699053b9" exitCode=0 Oct 30 21:50:57.762417 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:57.762387 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" event={"ID":"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03","Type":"ContainerDied","Data":"40ce500be9f3187e2ba92df7e74a72500a4159b6f14a949cf9f66610699053b9"} Oct 30 21:50:57.764088 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:57.764063 4335 generic.go:356] "Generic (PLEG): container finished" podID="b117cee9-ce1b-421a-bd91-1d70a0483047" containerID="8b78f50320bedc62ad6571ad275d537473a78da0ddcd8fd02f3cd55f7c63f4f8" exitCode=0 Oct 30 21:50:57.764191 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:57.764148 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" event={"ID":"b117cee9-ce1b-421a-bd91-1d70a0483047","Type":"ContainerDied","Data":"8b78f50320bedc62ad6571ad275d537473a78da0ddcd8fd02f3cd55f7c63f4f8"} Oct 30 21:50:57.766235 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:57.766200 4335 generic.go:356] "Generic (PLEG): container finished" podID="2a6e38bc-3e75-4392-90b2-a0472b37a443" containerID="d39d1f0a0ab1e54c9ac481003bca1eaa3f0004acb78a9f9ba890cb7e65d8ecf3" exitCode=0 Oct 30 21:50:57.766327 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:57.766234 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" event={"ID":"2a6e38bc-3e75-4392-90b2-a0472b37a443","Type":"ContainerDied","Data":"d39d1f0a0ab1e54c9ac481003bca1eaa3f0004acb78a9f9ba890cb7e65d8ecf3"} Oct 30 21:50:57.769049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:57.769025 4335 generic.go:356] "Generic (PLEG): container finished" podID="cd8714dd-e64c-440b-b110-6a2d087e5233" containerID="a3ca50ab4f74073515a26023d7064d18afa384cb1e2af1e8ab84d8f88bd82b64" exitCode=0 Oct 30 21:50:57.769308 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:57.769260 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4" event={"ID":"cd8714dd-e64c-440b-b110-6a2d087e5233","Type":"ContainerDied","Data":"a3ca50ab4f74073515a26023d7064d18afa384cb1e2af1e8ab84d8f88bd82b64"} Oct 30 21:50:57.771459 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:57.771432 4335 generic.go:356] "Generic (PLEG): container finished" podID="b128ca38-8f49-4584-a0d0-f87fdc422db6" containerID="089456ffef9676d66a856bdf5b061064746cd553d2c8d3acfa3e5d55c425a591" exitCode=0 Oct 30 21:50:57.771626 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:57.771588 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd" event={"ID":"b128ca38-8f49-4584-a0d0-f87fdc422db6","Type":"ContainerDied","Data":"089456ffef9676d66a856bdf5b061064746cd553d2c8d3acfa3e5d55c425a591"} Oct 30 21:50:58.780026 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:58.779995 4335 generic.go:356] "Generic (PLEG): container finished" podID="0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03" containerID="5588fa0ae9ce0e1c1c58cecb7a8e6b183633b7e3f124fcd0b7fbf50eb76ef091" exitCode=0 Oct 30 21:50:58.780556 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:58.780088 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" event={"ID":"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03","Type":"ContainerDied","Data":"5588fa0ae9ce0e1c1c58cecb7a8e6b183633b7e3f124fcd0b7fbf50eb76ef091"} Oct 30 21:50:58.783310 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:58.783282 4335 generic.go:356] "Generic (PLEG): container finished" podID="84db9e63-3579-46d7-a274-96e919a39275" containerID="e7191e9c80afa29976644fb4dda025ab859c41a6834553f569d44d12321399ce" exitCode=0 Oct 30 21:50:58.783473 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:58.783378 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" event={"ID":"84db9e63-3579-46d7-a274-96e919a39275","Type":"ContainerDied","Data":"e7191e9c80afa29976644fb4dda025ab859c41a6834553f569d44d12321399ce"} Oct 30 21:50:58.787203 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:58.787113 4335 generic.go:356] "Generic (PLEG): container finished" podID="7e135608-c576-4f15-a0ae-6ebece176771" containerID="f1a657055014d59af3184bfac35b698a126b6a5cd774c494ef7e2abf4aa0e3da" exitCode=0 Oct 30 21:50:58.787203 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:58.787159 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" event={"ID":"7e135608-c576-4f15-a0ae-6ebece176771","Type":"ContainerDied","Data":"f1a657055014d59af3184bfac35b698a126b6a5cd774c494ef7e2abf4aa0e3da"} Oct 30 21:50:58.791294 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:58.791201 4335 generic.go:356] "Generic (PLEG): container finished" podID="04670690-bbaa-4329-983b-9dcb81ada2c7" containerID="8bbf28c0ea3683b2a6dfe6ee85b67c9e7f0addf66db0c89d1cdcb72329dc6c6a" exitCode=0 Oct 30 21:50:58.791294 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:58.791255 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm" event={"ID":"04670690-bbaa-4329-983b-9dcb81ada2c7","Type":"ContainerDied","Data":"8bbf28c0ea3683b2a6dfe6ee85b67c9e7f0addf66db0c89d1cdcb72329dc6c6a"} Oct 30 21:50:59.804546 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:59.804507 4335 generic.go:356] "Generic (PLEG): container finished" podID="b117cee9-ce1b-421a-bd91-1d70a0483047" containerID="8cb7f92d2f094d0950111c30d8f35a85d7738d5eed0e77f7c4555e2a57703b7e" exitCode=0 Oct 30 21:50:59.805046 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:59.804659 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" event={"ID":"b117cee9-ce1b-421a-bd91-1d70a0483047","Type":"ContainerDied","Data":"8cb7f92d2f094d0950111c30d8f35a85d7738d5eed0e77f7c4555e2a57703b7e"} Oct 30 21:50:59.810464 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:59.809897 4335 generic.go:356] "Generic (PLEG): container finished" podID="2a6e38bc-3e75-4392-90b2-a0472b37a443" containerID="e18750b655046cbb94caa6827254f0a76df380cfc5bcb2d16e8ff8f834078f38" exitCode=0 Oct 30 21:50:59.810464 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:59.809978 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" event={"ID":"2a6e38bc-3e75-4392-90b2-a0472b37a443","Type":"ContainerDied","Data":"e18750b655046cbb94caa6827254f0a76df380cfc5bcb2d16e8ff8f834078f38"} Oct 30 21:50:59.815284 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:59.815258 4335 generic.go:356] "Generic (PLEG): container finished" podID="b128ca38-8f49-4584-a0d0-f87fdc422db6" containerID="2155e8c9326764fd02e28ea07f0640d9d5aece2f3d4002146ccf92567ab09d07" exitCode=0 Oct 30 21:50:59.815408 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:59.815390 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd" event={"ID":"b128ca38-8f49-4584-a0d0-f87fdc422db6","Type":"ContainerDied","Data":"2155e8c9326764fd02e28ea07f0640d9d5aece2f3d4002146ccf92567ab09d07"} Oct 30 21:50:59.820594 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:59.820069 4335 generic.go:356] "Generic (PLEG): container finished" podID="f4ca5459-710c-48ee-9907-b9f1c0fb5fd7" containerID="111979c8a519b74696d9488216866a8d4da38d90e37aee415b6eb1365b0328a8" exitCode=0 Oct 30 21:50:59.820594 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:50:59.820105 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" event={"ID":"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7","Type":"ContainerDied","Data":"111979c8a519b74696d9488216866a8d4da38d90e37aee415b6eb1365b0328a8"} Oct 30 21:51:01.862529 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:01.862487 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" event={"ID":"140b47d4-0e7f-4e84-9d70-eaa209d01605","Type":"ContainerStarted","Data":"a54f07f527dd688380180db5fbb983b59b9d66a047f7cdcc316be0b87c13d472"} Oct 30 21:51:02.907231 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:02.906132 4335 generic.go:356] "Generic (PLEG): container finished" podID="140b47d4-0e7f-4e84-9d70-eaa209d01605" containerID="a54f07f527dd688380180db5fbb983b59b9d66a047f7cdcc316be0b87c13d472" exitCode=0 Oct 30 21:51:02.907231 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:02.906280 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" event={"ID":"140b47d4-0e7f-4e84-9d70-eaa209d01605","Type":"ContainerDied","Data":"a54f07f527dd688380180db5fbb983b59b9d66a047f7cdcc316be0b87c13d472"} Oct 30 21:51:06.957964 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:06.956939 4335 generic.go:356] "Generic (PLEG): container finished" podID="cd8714dd-e64c-440b-b110-6a2d087e5233" containerID="354cc69f996131c33de209bffe41d709f0b07239d07d1c41809ad1a26dea93c1" exitCode=0 Oct 30 21:51:06.957964 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:06.956997 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4" event={"ID":"cd8714dd-e64c-440b-b110-6a2d087e5233","Type":"ContainerDied","Data":"354cc69f996131c33de209bffe41d709f0b07239d07d1c41809ad1a26dea93c1"} Oct 30 21:51:16.084677 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:16.084136 4335 generic.go:356] "Generic (PLEG): container finished" podID="00124fd7-ac22-42ca-8fad-0047c9765e88" containerID="a94a19fcb8f8691b429df269a0aeb381a19bdde2104217ff1b6d94da01415ba8" exitCode=0 Oct 30 21:51:16.084677 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:16.084258 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84" event={"ID":"00124fd7-ac22-42ca-8fad-0047c9765e88","Type":"ContainerDied","Data":"a94a19fcb8f8691b429df269a0aeb381a19bdde2104217ff1b6d94da01415ba8"} Oct 30 21:51:17.372502 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:17.372429 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84" Oct 30 21:51:17.526758 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:17.526723 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00124fd7-ac22-42ca-8fad-0047c9765e88-kube-api-access-c8lc5\" (UniqueName: \"kubernetes.io/projected/00124fd7-ac22-42ca-8fad-0047c9765e88-kube-api-access-c8lc5\") pod \"00124fd7-ac22-42ca-8fad-0047c9765e88\" (UID: \"00124fd7-ac22-42ca-8fad-0047c9765e88\") " Oct 30 21:51:17.526948 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:17.526821 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00124fd7-ac22-42ca-8fad-0047c9765e88-util\" (UniqueName: \"kubernetes.io/empty-dir/00124fd7-ac22-42ca-8fad-0047c9765e88-util\") pod \"00124fd7-ac22-42ca-8fad-0047c9765e88\" (UID: \"00124fd7-ac22-42ca-8fad-0047c9765e88\") " Oct 30 21:51:17.526948 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:17.526871 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00124fd7-ac22-42ca-8fad-0047c9765e88-bundle\" (UniqueName: \"kubernetes.io/empty-dir/00124fd7-ac22-42ca-8fad-0047c9765e88-bundle\") pod \"00124fd7-ac22-42ca-8fad-0047c9765e88\" (UID: \"00124fd7-ac22-42ca-8fad-0047c9765e88\") " Oct 30 21:51:17.527764 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:17.527735 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00124fd7-ac22-42ca-8fad-0047c9765e88-bundle" pod "00124fd7-ac22-42ca-8fad-0047c9765e88" (UID: "00124fd7-ac22-42ca-8fad-0047c9765e88"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:17.533707 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:17.533679 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00124fd7-ac22-42ca-8fad-0047c9765e88-util" pod "00124fd7-ac22-42ca-8fad-0047c9765e88" (UID: "00124fd7-ac22-42ca-8fad-0047c9765e88"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:17.552229 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:17.552189 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00124fd7-ac22-42ca-8fad-0047c9765e88-kube-api-access-c8lc5" pod "00124fd7-ac22-42ca-8fad-0047c9765e88" (UID: "00124fd7-ac22-42ca-8fad-0047c9765e88"). InnerVolumeSpecName "kube-api-access-c8lc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:17.628084 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:17.627907 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8lc5\" (UniqueName: \"kubernetes.io/projected/00124fd7-ac22-42ca-8fad-0047c9765e88-kube-api-access-c8lc5\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:17.628084 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:17.627941 4335 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00124fd7-ac22-42ca-8fad-0047c9765e88-util\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:17.628084 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:17.627957 4335 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00124fd7-ac22-42ca-8fad-0047c9765e88-bundle\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:18.113401 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:18.112051 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84" event={"ID":"00124fd7-ac22-42ca-8fad-0047c9765e88","Type":"ContainerDied","Data":"4113dbdd78d7fb011ec81925a2691bd967b0aa46565d3653eac7939718548a72"} Oct 30 21:51:18.113401 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:18.112109 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4113dbdd78d7fb011ec81925a2691bd967b0aa46565d3653eac7939718548a72" Oct 30 21:51:18.113401 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:18.112201 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71s5b84" Oct 30 21:51:19.123065 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:19.123003 4335 generic.go:356] "Generic (PLEG): container finished" podID="140b47d4-0e7f-4e84-9d70-eaa209d01605" containerID="0b7bc22dee7150768ec527adaf0681a254ef7cab262ef0f64b86f3c6a79487bd" exitCode=0 Oct 30 21:51:19.123534 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:19.123152 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" event={"ID":"140b47d4-0e7f-4e84-9d70-eaa209d01605","Type":"ContainerDied","Data":"0b7bc22dee7150768ec527adaf0681a254ef7cab262ef0f64b86f3c6a79487bd"} Oct 30 21:51:19.127359 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:19.127331 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" event={"ID":"cca7fe9d-c55b-4597-843d-a9cf78efc242","Type":"ContainerStarted","Data":"c42e5c62a64918cc690d5a2394689c09549dd6344fea0c59bfc573f22dce1def"} Oct 30 21:51:19.361996 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:19.361873 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" podStartSLOduration=4.322126504 podStartE2EDuration="42.361852839s" totalImagesPullingTime="38.039726335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:19.360848049 +0000 UTC m=+412.461279781" watchObservedRunningTime="2026-10-30 21:51:19.361852839 +0000 UTC m=+412.462284572" Oct 30 21:51:20.145231 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:20.145198 4335 generic.go:356] "Generic (PLEG): container finished" podID="cca7fe9d-c55b-4597-843d-a9cf78efc242" containerID="c42e5c62a64918cc690d5a2394689c09549dd6344fea0c59bfc573f22dce1def" exitCode=0 Oct 30 21:51:20.145732 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:20.145717 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" event={"ID":"cca7fe9d-c55b-4597-843d-a9cf78efc242","Type":"ContainerDied","Data":"c42e5c62a64918cc690d5a2394689c09549dd6344fea0c59bfc573f22dce1def"} Oct 30 21:51:20.375796 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:20.375768 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" Oct 30 21:51:20.565627 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:20.565577 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/140b47d4-0e7f-4e84-9d70-eaa209d01605-util\" (UniqueName: \"kubernetes.io/empty-dir/140b47d4-0e7f-4e84-9d70-eaa209d01605-util\") pod \"140b47d4-0e7f-4e84-9d70-eaa209d01605\" (UID: \"140b47d4-0e7f-4e84-9d70-eaa209d01605\") " Oct 30 21:51:20.565812 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:20.565711 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/140b47d4-0e7f-4e84-9d70-eaa209d01605-bundle\" (UniqueName: \"kubernetes.io/empty-dir/140b47d4-0e7f-4e84-9d70-eaa209d01605-bundle\") pod \"140b47d4-0e7f-4e84-9d70-eaa209d01605\" (UID: \"140b47d4-0e7f-4e84-9d70-eaa209d01605\") " Oct 30 21:51:20.565812 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:20.565755 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/140b47d4-0e7f-4e84-9d70-eaa209d01605-kube-api-access-p6hf7\" (UniqueName: \"kubernetes.io/projected/140b47d4-0e7f-4e84-9d70-eaa209d01605-kube-api-access-p6hf7\") pod \"140b47d4-0e7f-4e84-9d70-eaa209d01605\" (UID: \"140b47d4-0e7f-4e84-9d70-eaa209d01605\") " Oct 30 21:51:20.567167 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:20.567104 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140b47d4-0e7f-4e84-9d70-eaa209d01605-bundle" pod "140b47d4-0e7f-4e84-9d70-eaa209d01605" (UID: "140b47d4-0e7f-4e84-9d70-eaa209d01605"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:20.571950 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:20.571920 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140b47d4-0e7f-4e84-9d70-eaa209d01605-kube-api-access-p6hf7" pod "140b47d4-0e7f-4e84-9d70-eaa209d01605" (UID: "140b47d4-0e7f-4e84-9d70-eaa209d01605"). InnerVolumeSpecName "kube-api-access-p6hf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:20.574734 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:20.574710 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140b47d4-0e7f-4e84-9d70-eaa209d01605-util" pod "140b47d4-0e7f-4e84-9d70-eaa209d01605" (UID: "140b47d4-0e7f-4e84-9d70-eaa209d01605"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:20.666821 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:20.666785 4335 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/140b47d4-0e7f-4e84-9d70-eaa209d01605-bundle\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:20.666821 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:20.666824 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6hf7\" (UniqueName: \"kubernetes.io/projected/140b47d4-0e7f-4e84-9d70-eaa209d01605-kube-api-access-p6hf7\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:20.667069 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:20.666838 4335 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/140b47d4-0e7f-4e84-9d70-eaa209d01605-util\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.170321 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:21.170285 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" Oct 30 21:51:21.170771 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:21.170749 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp7gl2" event={"ID":"140b47d4-0e7f-4e84-9d70-eaa209d01605","Type":"ContainerDied","Data":"3e998bc1d2a9c179398d9d41bd442f9d03dd8e202d071c926888dcfc704384cd"} Oct 30 21:51:21.170845 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:21.170793 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e998bc1d2a9c179398d9d41bd442f9d03dd8e202d071c926888dcfc704384cd" Oct 30 21:51:21.377267 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:21.377236 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" Oct 30 21:51:21.475042 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:21.475008 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca7fe9d-c55b-4597-843d-a9cf78efc242-kube-api-access-qk6cb\" (UniqueName: \"kubernetes.io/projected/cca7fe9d-c55b-4597-843d-a9cf78efc242-kube-api-access-qk6cb\") pod \"cca7fe9d-c55b-4597-843d-a9cf78efc242\" (UID: \"cca7fe9d-c55b-4597-843d-a9cf78efc242\") " Oct 30 21:51:21.475236 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:21.475076 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca7fe9d-c55b-4597-843d-a9cf78efc242-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cca7fe9d-c55b-4597-843d-a9cf78efc242-bundle\") pod \"cca7fe9d-c55b-4597-843d-a9cf78efc242\" (UID: \"cca7fe9d-c55b-4597-843d-a9cf78efc242\") " Oct 30 21:51:21.475236 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:21.475113 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca7fe9d-c55b-4597-843d-a9cf78efc242-util\" (UniqueName: \"kubernetes.io/empty-dir/cca7fe9d-c55b-4597-843d-a9cf78efc242-util\") pod \"cca7fe9d-c55b-4597-843d-a9cf78efc242\" (UID: \"cca7fe9d-c55b-4597-843d-a9cf78efc242\") " Oct 30 21:51:21.476585 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:21.476559 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca7fe9d-c55b-4597-843d-a9cf78efc242-bundle" pod "cca7fe9d-c55b-4597-843d-a9cf78efc242" (UID: "cca7fe9d-c55b-4597-843d-a9cf78efc242"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:21.481395 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:21.481368 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca7fe9d-c55b-4597-843d-a9cf78efc242-util" pod "cca7fe9d-c55b-4597-843d-a9cf78efc242" (UID: "cca7fe9d-c55b-4597-843d-a9cf78efc242"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:21.487812 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:21.487785 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca7fe9d-c55b-4597-843d-a9cf78efc242-kube-api-access-qk6cb" pod "cca7fe9d-c55b-4597-843d-a9cf78efc242" (UID: "cca7fe9d-c55b-4597-843d-a9cf78efc242"). InnerVolumeSpecName "kube-api-access-qk6cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:21.576404 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:21.576318 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk6cb\" (UniqueName: \"kubernetes.io/projected/cca7fe9d-c55b-4597-843d-a9cf78efc242-kube-api-access-qk6cb\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.576404 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:21.576352 4335 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cca7fe9d-c55b-4597-843d-a9cf78efc242-bundle\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.576404 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:21.576369 4335 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cca7fe9d-c55b-4597-843d-a9cf78efc242-util\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.190647 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:22.190309 4335 generic.go:356] "Generic (PLEG): container finished" podID="cd8714dd-e64c-440b-b110-6a2d087e5233" containerID="11dfb40dfee997d1935ec9615d8ec71482e6e16ab4bc25b03410a2483757c9fa" exitCode=0 Oct 30 21:51:22.190647 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:22.190390 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4" event={"ID":"cd8714dd-e64c-440b-b110-6a2d087e5233","Type":"ContainerDied","Data":"11dfb40dfee997d1935ec9615d8ec71482e6e16ab4bc25b03410a2483757c9fa"} Oct 30 21:51:22.195798 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:22.193856 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" event={"ID":"cca7fe9d-c55b-4597-843d-a9cf78efc242","Type":"ContainerDied","Data":"ba70986d22790071803a7858e190b0c897caf5998aab21f41fce27ed9d8c1b50"} Oct 30 21:51:22.195798 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:22.193898 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba70986d22790071803a7858e190b0c897caf5998aab21f41fce27ed9d8c1b50" Oct 30 21:51:22.195798 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:22.193971 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fkh66" Oct 30 21:51:23.208907 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.208870 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" event={"ID":"2a6e38bc-3e75-4392-90b2-a0472b37a443","Type":"ContainerStarted","Data":"1431ad04ff5bdb52292467f0a463df64f1bab8fb45b4224296cda85b70a039d5"} Oct 30 21:51:23.213891 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.213801 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" event={"ID":"84db9e63-3579-46d7-a274-96e919a39275","Type":"ContainerStarted","Data":"d2fb385163e2a5d701fe907e009d00c10142916c034ecb360a7ca01e5e362579"} Oct 30 21:51:23.221892 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.221861 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" event={"ID":"7e135608-c576-4f15-a0ae-6ebece176771","Type":"ContainerStarted","Data":"2b8511dbe4b0b9af2b391f2f15fc25eaff2f5944a55513f90b484b08db120ed3"} Oct 30 21:51:23.225323 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.225294 4335 generic.go:356] "Generic (PLEG): container finished" podID="04670690-bbaa-4329-983b-9dcb81ada2c7" containerID="8f893123217b6ae2b0c4d3f636048e5a6a6daa2f5077578d7d2bfae0f0cdcf6c" exitCode=0 Oct 30 21:51:23.225421 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.225389 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm" event={"ID":"04670690-bbaa-4329-983b-9dcb81ada2c7","Type":"ContainerDied","Data":"8f893123217b6ae2b0c4d3f636048e5a6a6daa2f5077578d7d2bfae0f0cdcf6c"} Oct 30 21:51:23.231165 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.231144 4335 generic.go:356] "Generic (PLEG): container finished" podID="0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03" containerID="8d44106e7e7c97ae5611155573bff00afc41ade126003a752f3885c9f35512de" exitCode=0 Oct 30 21:51:23.231288 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.231274 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" event={"ID":"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03","Type":"ContainerDied","Data":"8d44106e7e7c97ae5611155573bff00afc41ade126003a752f3885c9f35512de"} Oct 30 21:51:23.234591 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.234571 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" event={"ID":"b117cee9-ce1b-421a-bd91-1d70a0483047","Type":"ContainerStarted","Data":"257851f45e211de01988ff69fa54028549cfef88ac8b1c0310660bbe23b8ff8d"} Oct 30 21:51:23.327233 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.327139 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" podStartSLOduration=3.810915818 podStartE2EDuration="43.327119079s" totalImagesPullingTime="39.516203261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:23.325278383 +0000 UTC m=+416.425710114" watchObservedRunningTime="2026-10-30 21:51:23.327119079 +0000 UTC m=+416.427550812" Oct 30 21:51:23.427560 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.427540 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4" Oct 30 21:51:23.444739 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.444697 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" podStartSLOduration=3.591544663 podStartE2EDuration="43.444676604s" totalImagesPullingTime="39.853131941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:23.443171326 +0000 UTC m=+416.543603059" watchObservedRunningTime="2026-10-30 21:51:23.444676604 +0000 UTC m=+416.545108336" Oct 30 21:51:23.559563 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.559510 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" podStartSLOduration=2.527715756 podStartE2EDuration="50.559494193s" totalImagesPullingTime="48.031778437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:23.559314611 +0000 UTC m=+416.659746356" watchObservedRunningTime="2026-10-30 21:51:23.559494193 +0000 UTC m=+416.659925928" Oct 30 21:51:23.594577 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.594551 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8714dd-e64c-440b-b110-6a2d087e5233-util\" (UniqueName: \"kubernetes.io/empty-dir/cd8714dd-e64c-440b-b110-6a2d087e5233-util\") pod \"cd8714dd-e64c-440b-b110-6a2d087e5233\" (UID: \"cd8714dd-e64c-440b-b110-6a2d087e5233\") " Oct 30 21:51:23.594688 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.594671 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd8714dd-e64c-440b-b110-6a2d087e5233-kube-api-access-jkqdq\" (UniqueName: \"kubernetes.io/projected/cd8714dd-e64c-440b-b110-6a2d087e5233-kube-api-access-jkqdq\") pod \"cd8714dd-e64c-440b-b110-6a2d087e5233\" (UID: \"cd8714dd-e64c-440b-b110-6a2d087e5233\") " Oct 30 21:51:23.594772 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.594728 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8714dd-e64c-440b-b110-6a2d087e5233-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd8714dd-e64c-440b-b110-6a2d087e5233-bundle\") pod \"cd8714dd-e64c-440b-b110-6a2d087e5233\" (UID: \"cd8714dd-e64c-440b-b110-6a2d087e5233\") " Oct 30 21:51:23.595070 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.595048 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8714dd-e64c-440b-b110-6a2d087e5233-bundle" pod "cd8714dd-e64c-440b-b110-6a2d087e5233" (UID: "cd8714dd-e64c-440b-b110-6a2d087e5233"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:23.596851 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.596823 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8714dd-e64c-440b-b110-6a2d087e5233-kube-api-access-jkqdq" pod "cd8714dd-e64c-440b-b110-6a2d087e5233" (UID: "cd8714dd-e64c-440b-b110-6a2d087e5233"). InnerVolumeSpecName "kube-api-access-jkqdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:23.598638 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.598593 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8714dd-e64c-440b-b110-6a2d087e5233-util" pod "cd8714dd-e64c-440b-b110-6a2d087e5233" (UID: "cd8714dd-e64c-440b-b110-6a2d087e5233"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:23.677029 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.676989 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" podStartSLOduration=2.689078364 podStartE2EDuration="50.676976564s" totalImagesPullingTime="47.9878982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:23.675756556 +0000 UTC m=+416.776188290" watchObservedRunningTime="2026-10-30 21:51:23.676976564 +0000 UTC m=+416.777408296" Oct 30 21:51:23.695985 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.695958 4335 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd8714dd-e64c-440b-b110-6a2d087e5233-util\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:23.696060 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.695984 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkqdq\" (UniqueName: \"kubernetes.io/projected/cd8714dd-e64c-440b-b110-6a2d087e5233-kube-api-access-jkqdq\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:23.696060 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.696000 4335 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd8714dd-e64c-440b-b110-6a2d087e5233-bundle\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:23.793131 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:23.793031 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" podStartSLOduration=4.272330625 podStartE2EDuration="51.793017035s" totalImagesPullingTime="47.52068641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:23.792926274 +0000 UTC m=+416.893358008" watchObservedRunningTime="2026-10-30 21:51:23.793017035 +0000 UTC m=+416.893448767" Oct 30 21:51:24.240876 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.240846 4335 generic.go:356] "Generic (PLEG): container finished" podID="b117cee9-ce1b-421a-bd91-1d70a0483047" containerID="257851f45e211de01988ff69fa54028549cfef88ac8b1c0310660bbe23b8ff8d" exitCode=0 Oct 30 21:51:24.241280 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.240932 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" event={"ID":"b117cee9-ce1b-421a-bd91-1d70a0483047","Type":"ContainerDied","Data":"257851f45e211de01988ff69fa54028549cfef88ac8b1c0310660bbe23b8ff8d"} Oct 30 21:51:24.242504 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.242487 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 21:51:24.243163 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.243142 4335 generic.go:356] "Generic (PLEG): container finished" podID="2a6e38bc-3e75-4392-90b2-a0472b37a443" containerID="1431ad04ff5bdb52292467f0a463df64f1bab8fb45b4224296cda85b70a039d5" exitCode=1 Oct 30 21:51:24.243249 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.243207 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" event={"ID":"2a6e38bc-3e75-4392-90b2-a0472b37a443","Type":"ContainerDied","Data":"1431ad04ff5bdb52292467f0a463df64f1bab8fb45b4224296cda85b70a039d5"} Oct 30 21:51:24.245054 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.245034 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4" event={"ID":"cd8714dd-e64c-440b-b110-6a2d087e5233","Type":"ContainerDied","Data":"c105a8ed4a3f33fb2c9dfd58b44c3cc4b0d46d7c924f17d6f8cb1c52f81797e5"} Oct 30 21:51:24.245151 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.245071 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c105a8ed4a3f33fb2c9dfd58b44c3cc4b0d46d7c924f17d6f8cb1c52f81797e5" Oct 30 21:51:24.245151 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.245037 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qw4p4" Oct 30 21:51:24.247114 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.247097 4335 generic.go:356] "Generic (PLEG): container finished" podID="b128ca38-8f49-4584-a0d0-f87fdc422db6" containerID="491e16cb625261db00f9553238d8f2b97cffdf1d98c27190971d032a48864356" exitCode=0 Oct 30 21:51:24.247185 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.247169 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd" event={"ID":"b128ca38-8f49-4584-a0d0-f87fdc422db6","Type":"ContainerDied","Data":"491e16cb625261db00f9553238d8f2b97cffdf1d98c27190971d032a48864356"} Oct 30 21:51:24.249294 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.249274 4335 generic.go:356] "Generic (PLEG): container finished" podID="84db9e63-3579-46d7-a274-96e919a39275" containerID="d2fb385163e2a5d701fe907e009d00c10142916c034ecb360a7ca01e5e362579" exitCode=0 Oct 30 21:51:24.249391 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.249338 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" event={"ID":"84db9e63-3579-46d7-a274-96e919a39275","Type":"ContainerDied","Data":"d2fb385163e2a5d701fe907e009d00c10142916c034ecb360a7ca01e5e362579"} Oct 30 21:51:24.251368 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.251345 4335 generic.go:356] "Generic (PLEG): container finished" podID="7e135608-c576-4f15-a0ae-6ebece176771" containerID="2b8511dbe4b0b9af2b391f2f15fc25eaff2f5944a55513f90b484b08db120ed3" exitCode=0 Oct 30 21:51:24.251469 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.251427 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" event={"ID":"7e135608-c576-4f15-a0ae-6ebece176771","Type":"ContainerDied","Data":"2b8511dbe4b0b9af2b391f2f15fc25eaff2f5944a55513f90b484b08db120ed3"} Oct 30 21:51:24.253526 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.253505 4335 generic.go:356] "Generic (PLEG): container finished" podID="f4ca5459-710c-48ee-9907-b9f1c0fb5fd7" containerID="c9322921e040d4745b8b02af0b369fef3b2c2f9661b0a07ae11848d7c3eccf43" exitCode=0 Oct 30 21:51:24.253637 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.253562 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" event={"ID":"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7","Type":"ContainerDied","Data":"c9322921e040d4745b8b02af0b369fef3b2c2f9661b0a07ae11848d7c3eccf43"} Oct 30 21:51:24.389212 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.389188 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm" Oct 30 21:51:24.399388 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.399368 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" Oct 30 21:51:24.501568 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.501514 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04670690-bbaa-4329-983b-9dcb81ada2c7-util\" (UniqueName: \"kubernetes.io/empty-dir/04670690-bbaa-4329-983b-9dcb81ada2c7-util\") pod \"04670690-bbaa-4329-983b-9dcb81ada2c7\" (UID: \"04670690-bbaa-4329-983b-9dcb81ada2c7\") " Oct 30 21:51:24.501568 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.501550 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-kube-api-access-4ht57\" (UniqueName: \"kubernetes.io/projected/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-kube-api-access-4ht57\") pod \"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03\" (UID: \"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03\") " Oct 30 21:51:24.501711 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.501573 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04670690-bbaa-4329-983b-9dcb81ada2c7-kube-api-access-c98r9\" (UniqueName: \"kubernetes.io/projected/04670690-bbaa-4329-983b-9dcb81ada2c7-kube-api-access-c98r9\") pod \"04670690-bbaa-4329-983b-9dcb81ada2c7\" (UID: \"04670690-bbaa-4329-983b-9dcb81ada2c7\") " Oct 30 21:51:24.501711 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.501642 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04670690-bbaa-4329-983b-9dcb81ada2c7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/04670690-bbaa-4329-983b-9dcb81ada2c7-bundle\") pod \"04670690-bbaa-4329-983b-9dcb81ada2c7\" (UID: \"04670690-bbaa-4329-983b-9dcb81ada2c7\") " Oct 30 21:51:24.501711 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.501694 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-util\" (UniqueName: \"kubernetes.io/empty-dir/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-util\") pod \"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03\" (UID: \"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03\") " Oct 30 21:51:24.501847 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.501749 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-bundle\") pod \"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03\" (UID: \"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03\") " Oct 30 21:51:24.502182 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.502155 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04670690-bbaa-4329-983b-9dcb81ada2c7-bundle" pod "04670690-bbaa-4329-983b-9dcb81ada2c7" (UID: "04670690-bbaa-4329-983b-9dcb81ada2c7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:24.502293 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.502252 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-bundle" pod "0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03" (UID: "0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:24.503765 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.503736 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-kube-api-access-4ht57" pod "0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03" (UID: "0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03"). InnerVolumeSpecName "kube-api-access-4ht57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:24.504138 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.504124 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04670690-bbaa-4329-983b-9dcb81ada2c7-kube-api-access-c98r9" pod "04670690-bbaa-4329-983b-9dcb81ada2c7" (UID: "04670690-bbaa-4329-983b-9dcb81ada2c7"). InnerVolumeSpecName "kube-api-access-c98r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:24.507666 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.507647 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04670690-bbaa-4329-983b-9dcb81ada2c7-util" pod "04670690-bbaa-4329-983b-9dcb81ada2c7" (UID: "04670690-bbaa-4329-983b-9dcb81ada2c7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:24.508407 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.508386 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-util" pod "0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03" (UID: "0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:24.602362 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.602337 4335 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-util\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.602362 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.602358 4335 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-bundle\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.602476 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.602367 4335 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04670690-bbaa-4329-983b-9dcb81ada2c7-util\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.602476 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.602376 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ht57\" (UniqueName: \"kubernetes.io/projected/0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03-kube-api-access-4ht57\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.602476 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.602386 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c98r9\" (UniqueName: \"kubernetes.io/projected/04670690-bbaa-4329-983b-9dcb81ada2c7-kube-api-access-c98r9\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.602476 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:24.602395 4335 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04670690-bbaa-4329-983b-9dcb81ada2c7-bundle\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.260371 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.260344 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm" Oct 30 21:51:25.260371 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.260352 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xqbmm" event={"ID":"04670690-bbaa-4329-983b-9dcb81ada2c7","Type":"ContainerDied","Data":"4bda04f679d03f52e8c8516ea5d185e7eeda6dc5733027a491310c240946fcc7"} Oct 30 21:51:25.260874 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.260401 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bda04f679d03f52e8c8516ea5d185e7eeda6dc5733027a491310c240946fcc7" Oct 30 21:51:25.262727 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.262592 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" Oct 30 21:51:25.262727 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.262656 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66hzl7" event={"ID":"0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03","Type":"ContainerDied","Data":"d286b414b2ed97e34e612f807039597fd2a5ebc6cca445e2474bb669eebb0a21"} Oct 30 21:51:25.262727 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.262702 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d286b414b2ed97e34e612f807039597fd2a5ebc6cca445e2474bb669eebb0a21" Oct 30 21:51:25.350245 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:51:25.350195 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:51:25.421438 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.421415 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" Oct 30 21:51:25.479091 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.479061 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" Oct 30 21:51:25.483437 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.483417 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd" Oct 30 21:51:25.504579 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.504557 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" Oct 30 21:51:25.510775 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.510755 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 21:51:25.512336 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.512319 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" Oct 30 21:51:25.513354 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.513338 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84db9e63-3579-46d7-a274-96e919a39275-bundle\" (UniqueName: \"kubernetes.io/empty-dir/84db9e63-3579-46d7-a274-96e919a39275-bundle\") pod \"84db9e63-3579-46d7-a274-96e919a39275\" (UID: \"84db9e63-3579-46d7-a274-96e919a39275\") " Oct 30 21:51:25.513442 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.513365 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b128ca38-8f49-4584-a0d0-f87fdc422db6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b128ca38-8f49-4584-a0d0-f87fdc422db6-bundle\") pod \"b128ca38-8f49-4584-a0d0-f87fdc422db6\" (UID: \"b128ca38-8f49-4584-a0d0-f87fdc422db6\") " Oct 30 21:51:25.513442 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.513387 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b117cee9-ce1b-421a-bd91-1d70a0483047-util\" (UniqueName: \"kubernetes.io/empty-dir/b117cee9-ce1b-421a-bd91-1d70a0483047-util\") pod \"b117cee9-ce1b-421a-bd91-1d70a0483047\" (UID: \"b117cee9-ce1b-421a-bd91-1d70a0483047\") " Oct 30 21:51:25.513442 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.513405 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84db9e63-3579-46d7-a274-96e919a39275-util\" (UniqueName: \"kubernetes.io/empty-dir/84db9e63-3579-46d7-a274-96e919a39275-util\") pod \"84db9e63-3579-46d7-a274-96e919a39275\" (UID: \"84db9e63-3579-46d7-a274-96e919a39275\") " Oct 30 21:51:25.513442 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.513426 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e135608-c576-4f15-a0ae-6ebece176771-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e135608-c576-4f15-a0ae-6ebece176771-bundle\") pod \"7e135608-c576-4f15-a0ae-6ebece176771\" (UID: \"7e135608-c576-4f15-a0ae-6ebece176771\") " Oct 30 21:51:25.513677 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.513461 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a6e38bc-3e75-4392-90b2-a0472b37a443-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a6e38bc-3e75-4392-90b2-a0472b37a443-bundle\") pod \"2a6e38bc-3e75-4392-90b2-a0472b37a443\" (UID: \"2a6e38bc-3e75-4392-90b2-a0472b37a443\") " Oct 30 21:51:25.513677 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.513511 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84db9e63-3579-46d7-a274-96e919a39275-kube-api-access-ttcg9\" (UniqueName: \"kubernetes.io/projected/84db9e63-3579-46d7-a274-96e919a39275-kube-api-access-ttcg9\") pod \"84db9e63-3579-46d7-a274-96e919a39275\" (UID: \"84db9e63-3579-46d7-a274-96e919a39275\") " Oct 30 21:51:25.513677 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.513595 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e135608-c576-4f15-a0ae-6ebece176771-kube-api-access-vj7qg\" (UniqueName: \"kubernetes.io/projected/7e135608-c576-4f15-a0ae-6ebece176771-kube-api-access-vj7qg\") pod \"7e135608-c576-4f15-a0ae-6ebece176771\" (UID: \"7e135608-c576-4f15-a0ae-6ebece176771\") " Oct 30 21:51:25.513794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.513691 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b117cee9-ce1b-421a-bd91-1d70a0483047-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b117cee9-ce1b-421a-bd91-1d70a0483047-bundle\") pod \"b117cee9-ce1b-421a-bd91-1d70a0483047\" (UID: \"b117cee9-ce1b-421a-bd91-1d70a0483047\") " Oct 30 21:51:25.513794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.513720 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b128ca38-8f49-4584-a0d0-f87fdc422db6-kube-api-access-6x2rx\" (UniqueName: \"kubernetes.io/projected/b128ca38-8f49-4584-a0d0-f87fdc422db6-kube-api-access-6x2rx\") pod \"b128ca38-8f49-4584-a0d0-f87fdc422db6\" (UID: \"b128ca38-8f49-4584-a0d0-f87fdc422db6\") " Oct 30 21:51:25.513794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.513756 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b117cee9-ce1b-421a-bd91-1d70a0483047-kube-api-access-shk99\" (UniqueName: \"kubernetes.io/projected/b117cee9-ce1b-421a-bd91-1d70a0483047-kube-api-access-shk99\") pod \"b117cee9-ce1b-421a-bd91-1d70a0483047\" (UID: \"b117cee9-ce1b-421a-bd91-1d70a0483047\") " Oct 30 21:51:25.513950 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.513792 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a6e38bc-3e75-4392-90b2-a0472b37a443-kube-api-access-vfvwp\" (UniqueName: \"kubernetes.io/projected/2a6e38bc-3e75-4392-90b2-a0472b37a443-kube-api-access-vfvwp\") pod \"2a6e38bc-3e75-4392-90b2-a0472b37a443\" (UID: \"2a6e38bc-3e75-4392-90b2-a0472b37a443\") " Oct 30 21:51:25.513950 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.513823 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e135608-c576-4f15-a0ae-6ebece176771-util\" (UniqueName: \"kubernetes.io/empty-dir/7e135608-c576-4f15-a0ae-6ebece176771-util\") pod \"7e135608-c576-4f15-a0ae-6ebece176771\" (UID: \"7e135608-c576-4f15-a0ae-6ebece176771\") " Oct 30 21:51:25.513950 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.513867 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b128ca38-8f49-4584-a0d0-f87fdc422db6-util\" (UniqueName: \"kubernetes.io/empty-dir/b128ca38-8f49-4584-a0d0-f87fdc422db6-util\") pod \"b128ca38-8f49-4584-a0d0-f87fdc422db6\" (UID: \"b128ca38-8f49-4584-a0d0-f87fdc422db6\") " Oct 30 21:51:25.513950 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.513915 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a6e38bc-3e75-4392-90b2-a0472b37a443-util\" (UniqueName: \"kubernetes.io/empty-dir/2a6e38bc-3e75-4392-90b2-a0472b37a443-util\") pod \"2a6e38bc-3e75-4392-90b2-a0472b37a443\" (UID: \"2a6e38bc-3e75-4392-90b2-a0472b37a443\") " Oct 30 21:51:25.514148 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.514089 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e135608-c576-4f15-a0ae-6ebece176771-bundle" pod "7e135608-c576-4f15-a0ae-6ebece176771" (UID: "7e135608-c576-4f15-a0ae-6ebece176771"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:25.514551 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.514168 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6e38bc-3e75-4392-90b2-a0472b37a443-bundle" pod "2a6e38bc-3e75-4392-90b2-a0472b37a443" (UID: "2a6e38bc-3e75-4392-90b2-a0472b37a443"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:25.515214 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.514892 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b117cee9-ce1b-421a-bd91-1d70a0483047-bundle" pod "b117cee9-ce1b-421a-bd91-1d70a0483047" (UID: "b117cee9-ce1b-421a-bd91-1d70a0483047"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:25.515650 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.515580 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b128ca38-8f49-4584-a0d0-f87fdc422db6-bundle" pod "b128ca38-8f49-4584-a0d0-f87fdc422db6" (UID: "b128ca38-8f49-4584-a0d0-f87fdc422db6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:25.516480 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.516456 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b117cee9-ce1b-421a-bd91-1d70a0483047-kube-api-access-shk99" pod "b117cee9-ce1b-421a-bd91-1d70a0483047" (UID: "b117cee9-ce1b-421a-bd91-1d70a0483047"). InnerVolumeSpecName "kube-api-access-shk99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:25.519368 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.519336 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84db9e63-3579-46d7-a274-96e919a39275-bundle" pod "84db9e63-3579-46d7-a274-96e919a39275" (UID: "84db9e63-3579-46d7-a274-96e919a39275"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:25.520010 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.519989 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" Oct 30 21:51:25.520776 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.520742 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84db9e63-3579-46d7-a274-96e919a39275-kube-api-access-ttcg9" pod "84db9e63-3579-46d7-a274-96e919a39275" (UID: "84db9e63-3579-46d7-a274-96e919a39275"). InnerVolumeSpecName "kube-api-access-ttcg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:25.521812 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.521789 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e135608-c576-4f15-a0ae-6ebece176771-util" pod "7e135608-c576-4f15-a0ae-6ebece176771" (UID: "7e135608-c576-4f15-a0ae-6ebece176771"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:25.523270 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.523250 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b117cee9-ce1b-421a-bd91-1d70a0483047-util" pod "b117cee9-ce1b-421a-bd91-1d70a0483047" (UID: "b117cee9-ce1b-421a-bd91-1d70a0483047"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:25.524506 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.524483 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84db9e63-3579-46d7-a274-96e919a39275-util" pod "84db9e63-3579-46d7-a274-96e919a39275" (UID: "84db9e63-3579-46d7-a274-96e919a39275"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:25.524875 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.524849 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b128ca38-8f49-4584-a0d0-f87fdc422db6-kube-api-access-6x2rx" pod "b128ca38-8f49-4584-a0d0-f87fdc422db6" (UID: "b128ca38-8f49-4584-a0d0-f87fdc422db6"). InnerVolumeSpecName "kube-api-access-6x2rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:25.525042 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.525021 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b128ca38-8f49-4584-a0d0-f87fdc422db6-util" pod "b128ca38-8f49-4584-a0d0-f87fdc422db6" (UID: "b128ca38-8f49-4584-a0d0-f87fdc422db6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:25.525266 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.525244 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6e38bc-3e75-4392-90b2-a0472b37a443-kube-api-access-vfvwp" pod "2a6e38bc-3e75-4392-90b2-a0472b37a443" (UID: "2a6e38bc-3e75-4392-90b2-a0472b37a443"). InnerVolumeSpecName "kube-api-access-vfvwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:25.525371 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.525356 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6e38bc-3e75-4392-90b2-a0472b37a443-util" pod "2a6e38bc-3e75-4392-90b2-a0472b37a443" (UID: "2a6e38bc-3e75-4392-90b2-a0472b37a443"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:25.526145 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.526128 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e135608-c576-4f15-a0ae-6ebece176771-kube-api-access-vj7qg" pod "7e135608-c576-4f15-a0ae-6ebece176771" (UID: "7e135608-c576-4f15-a0ae-6ebece176771"). InnerVolumeSpecName "kube-api-access-vj7qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:25.614820 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.614767 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-bundle\") pod \"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7\" (UID: \"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7\") " Oct 30 21:51:25.614820 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.614818 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-kube-api-access-5gpxv\" (UniqueName: \"kubernetes.io/projected/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-kube-api-access-5gpxv\") pod \"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7\" (UID: \"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7\") " Oct 30 21:51:25.614943 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.614837 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-util\" (UniqueName: \"kubernetes.io/empty-dir/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-util\") pod \"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7\" (UID: \"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7\") " Oct 30 21:51:25.614981 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.614963 4335 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a6e38bc-3e75-4392-90b2-a0472b37a443-bundle\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.614981 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.614974 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttcg9\" (UniqueName: \"kubernetes.io/projected/84db9e63-3579-46d7-a274-96e919a39275-kube-api-access-ttcg9\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.615083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.614984 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj7qg\" (UniqueName: \"kubernetes.io/projected/7e135608-c576-4f15-a0ae-6ebece176771-kube-api-access-vj7qg\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.615083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.614993 4335 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b117cee9-ce1b-421a-bd91-1d70a0483047-bundle\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.615083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.615006 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x2rx\" (UniqueName: \"kubernetes.io/projected/b128ca38-8f49-4584-a0d0-f87fdc422db6-kube-api-access-6x2rx\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.615083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.615020 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shk99\" (UniqueName: \"kubernetes.io/projected/b117cee9-ce1b-421a-bd91-1d70a0483047-kube-api-access-shk99\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.615083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.615032 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfvwp\" (UniqueName: \"kubernetes.io/projected/2a6e38bc-3e75-4392-90b2-a0472b37a443-kube-api-access-vfvwp\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.615083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.615040 4335 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e135608-c576-4f15-a0ae-6ebece176771-util\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.615083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.615048 4335 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b128ca38-8f49-4584-a0d0-f87fdc422db6-util\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.615083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.615055 4335 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a6e38bc-3e75-4392-90b2-a0472b37a443-util\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.615083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.615063 4335 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84db9e63-3579-46d7-a274-96e919a39275-bundle\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.615083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.615070 4335 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b128ca38-8f49-4584-a0d0-f87fdc422db6-bundle\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.615083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.615076 4335 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b117cee9-ce1b-421a-bd91-1d70a0483047-util\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.615083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.615083 4335 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84db9e63-3579-46d7-a274-96e919a39275-util\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.615609 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.615093 4335 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e135608-c576-4f15-a0ae-6ebece176771-bundle\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.615609 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.615453 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-bundle" pod "f4ca5459-710c-48ee-9907-b9f1c0fb5fd7" (UID: "f4ca5459-710c-48ee-9907-b9f1c0fb5fd7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:25.616850 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.616831 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-kube-api-access-5gpxv" pod "f4ca5459-710c-48ee-9907-b9f1c0fb5fd7" (UID: "f4ca5459-710c-48ee-9907-b9f1c0fb5fd7"). InnerVolumeSpecName "kube-api-access-5gpxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:25.621049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.621028 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-util" pod "f4ca5459-710c-48ee-9907-b9f1c0fb5fd7" (UID: "f4ca5459-710c-48ee-9907-b9f1c0fb5fd7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:25.716340 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.716316 4335 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-bundle\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.716340 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.716338 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gpxv\" (UniqueName: \"kubernetes.io/projected/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-kube-api-access-5gpxv\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.716458 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:25.716352 4335 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4ca5459-710c-48ee-9907-b9f1c0fb5fd7-util\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:26.269287 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.269263 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" Oct 30 21:51:26.269760 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.269284 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhrss" event={"ID":"b117cee9-ce1b-421a-bd91-1d70a0483047","Type":"ContainerDied","Data":"57802345b2d8b073a9193158cb049520111232605a6b65c737fe141e28aa4084"} Oct 30 21:51:26.269760 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.269329 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57802345b2d8b073a9193158cb049520111232605a6b65c737fe141e28aa4084" Oct 30 21:51:26.270593 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.270575 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 21:51:26.271411 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.271397 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" Oct 30 21:51:26.271477 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.271423 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx" event={"ID":"2a6e38bc-3e75-4392-90b2-a0472b37a443","Type":"ContainerDied","Data":"d7cf203381ff71c20947fbc460c579b55439ccdb7c77189506eafa565d22d304"} Oct 30 21:51:26.271477 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.271462 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7cf203381ff71c20947fbc460c579b55439ccdb7c77189506eafa565d22d304" Oct 30 21:51:26.273707 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.273685 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd" event={"ID":"b128ca38-8f49-4584-a0d0-f87fdc422db6","Type":"ContainerDied","Data":"7570d1a81cb3d33f55a46a03c85d5908b50c570d2ea8f66aaf871fbafa2e268b"} Oct 30 21:51:26.273833 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.273720 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7570d1a81cb3d33f55a46a03c85d5908b50c570d2ea8f66aaf871fbafa2e268b" Oct 30 21:51:26.273833 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.273692 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crntbd" Oct 30 21:51:26.276090 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.276068 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" Oct 30 21:51:26.276227 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.276066 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apbsfd" event={"ID":"84db9e63-3579-46d7-a274-96e919a39275","Type":"ContainerDied","Data":"54bc53c4585a4e277d759ff163d79932deed6bbe5ccb77453cac672bc547c801"} Oct 30 21:51:26.276227 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.276194 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54bc53c4585a4e277d759ff163d79932deed6bbe5ccb77453cac672bc547c801" Oct 30 21:51:26.278405 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.278384 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" Oct 30 21:51:26.278882 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.278387 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fgnx6" event={"ID":"7e135608-c576-4f15-a0ae-6ebece176771","Type":"ContainerDied","Data":"f1bc0e2a29215d42ea03742db7c0460f96e43e5b88e67375f387257422e4d648"} Oct 30 21:51:26.279009 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.278891 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1bc0e2a29215d42ea03742db7c0460f96e43e5b88e67375f387257422e4d648" Oct 30 21:51:26.280833 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.280813 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" event={"ID":"f4ca5459-710c-48ee-9907-b9f1c0fb5fd7","Type":"ContainerDied","Data":"60973549d9dc79956f724a48a55615a0e04efb7ebe1789b2219010285bf5c790"} Oct 30 21:51:26.280926 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.280851 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60973549d9dc79956f724a48a55615a0e04efb7ebe1789b2219010285bf5c790" Oct 30 21:51:26.280972 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:26.280934 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847assqqp" Oct 30 21:51:27.251192 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251161 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg"] Oct 30 21:51:27.251433 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251417 4335 cpu_manager.go:414] "containerMap: removing container" podUID="00124fd7-ac22-42ca-8fad-0047c9765e88" containerName="util" Oct 30 21:51:27.251489 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251436 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00124fd7-ac22-42ca-8fad-0047c9765e88" containerName="util" Oct 30 21:51:27.251489 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251446 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f4ca5459-710c-48ee-9907-b9f1c0fb5fd7" containerName="pull" Oct 30 21:51:27.251489 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251455 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ca5459-710c-48ee-9907-b9f1c0fb5fd7" containerName="pull" Oct 30 21:51:27.251489 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251464 4335 cpu_manager.go:414] "containerMap: removing container" podUID="84db9e63-3579-46d7-a274-96e919a39275" containerName="extract" Oct 30 21:51:27.251489 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251470 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84db9e63-3579-46d7-a274-96e919a39275" containerName="extract" Oct 30 21:51:27.251489 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251480 4335 cpu_manager.go:414] "containerMap: removing container" podUID="b117cee9-ce1b-421a-bd91-1d70a0483047" containerName="util" Oct 30 21:51:27.251489 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251485 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b117cee9-ce1b-421a-bd91-1d70a0483047" containerName="util" Oct 30 21:51:27.251489 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251492 4335 cpu_manager.go:414] "containerMap: removing container" podUID="7e135608-c576-4f15-a0ae-6ebece176771" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251496 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e135608-c576-4f15-a0ae-6ebece176771" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251506 4335 cpu_manager.go:414] "containerMap: removing container" podUID="04670690-bbaa-4329-983b-9dcb81ada2c7" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251510 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04670690-bbaa-4329-983b-9dcb81ada2c7" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251517 4335 cpu_manager.go:414] "containerMap: removing container" podUID="04670690-bbaa-4329-983b-9dcb81ada2c7" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251521 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04670690-bbaa-4329-983b-9dcb81ada2c7" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251530 4335 cpu_manager.go:414] "containerMap: removing container" podUID="b128ca38-8f49-4584-a0d0-f87fdc422db6" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251537 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b128ca38-8f49-4584-a0d0-f87fdc422db6" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251547 4335 cpu_manager.go:414] "containerMap: removing container" podUID="cd8714dd-e64c-440b-b110-6a2d087e5233" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251551 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd8714dd-e64c-440b-b110-6a2d087e5233" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251557 4335 cpu_manager.go:414] "containerMap: removing container" podUID="cca7fe9d-c55b-4597-843d-a9cf78efc242" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251561 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca7fe9d-c55b-4597-843d-a9cf78efc242" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251566 4335 cpu_manager.go:414] "containerMap: removing container" podUID="2a6e38bc-3e75-4392-90b2-a0472b37a443" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251570 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a6e38bc-3e75-4392-90b2-a0472b37a443" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251576 4335 cpu_manager.go:414] "containerMap: removing container" podUID="0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251580 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251585 4335 cpu_manager.go:414] "containerMap: removing container" podUID="b128ca38-8f49-4584-a0d0-f87fdc422db6" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251589 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b128ca38-8f49-4584-a0d0-f87fdc422db6" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251595 4335 cpu_manager.go:414] "containerMap: removing container" podUID="0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251599 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251606 4335 cpu_manager.go:414] "containerMap: removing container" podUID="00124fd7-ac22-42ca-8fad-0047c9765e88" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251627 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00124fd7-ac22-42ca-8fad-0047c9765e88" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251638 4335 cpu_manager.go:414] "containerMap: removing container" podUID="140b47d4-0e7f-4e84-9d70-eaa209d01605" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251643 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="140b47d4-0e7f-4e84-9d70-eaa209d01605" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251649 4335 cpu_manager.go:414] "containerMap: removing container" podUID="2a6e38bc-3e75-4392-90b2-a0472b37a443" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251653 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a6e38bc-3e75-4392-90b2-a0472b37a443" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251659 4335 cpu_manager.go:414] "containerMap: removing container" podUID="140b47d4-0e7f-4e84-9d70-eaa209d01605" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251663 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="140b47d4-0e7f-4e84-9d70-eaa209d01605" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251678 4335 cpu_manager.go:414] "containerMap: removing container" podUID="84db9e63-3579-46d7-a274-96e919a39275" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251683 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84db9e63-3579-46d7-a274-96e919a39275" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251689 4335 cpu_manager.go:414] "containerMap: removing container" podUID="140b47d4-0e7f-4e84-9d70-eaa209d01605" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251693 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="140b47d4-0e7f-4e84-9d70-eaa209d01605" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251700 4335 cpu_manager.go:414] "containerMap: removing container" podUID="00124fd7-ac22-42ca-8fad-0047c9765e88" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251704 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00124fd7-ac22-42ca-8fad-0047c9765e88" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251712 4335 cpu_manager.go:414] "containerMap: removing container" podUID="84db9e63-3579-46d7-a274-96e919a39275" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251716 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84db9e63-3579-46d7-a274-96e919a39275" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251720 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f4ca5459-710c-48ee-9907-b9f1c0fb5fd7" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251724 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ca5459-710c-48ee-9907-b9f1c0fb5fd7" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251731 4335 cpu_manager.go:414] "containerMap: removing container" podUID="7e135608-c576-4f15-a0ae-6ebece176771" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251735 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e135608-c576-4f15-a0ae-6ebece176771" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251741 4335 cpu_manager.go:414] "containerMap: removing container" podUID="b117cee9-ce1b-421a-bd91-1d70a0483047" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251748 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b117cee9-ce1b-421a-bd91-1d70a0483047" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251760 4335 cpu_manager.go:414] "containerMap: removing container" podUID="0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251765 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251773 4335 cpu_manager.go:414] "containerMap: removing container" podUID="cd8714dd-e64c-440b-b110-6a2d087e5233" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251777 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd8714dd-e64c-440b-b110-6a2d087e5233" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251785 4335 cpu_manager.go:414] "containerMap: removing container" podUID="cca7fe9d-c55b-4597-843d-a9cf78efc242" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251790 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca7fe9d-c55b-4597-843d-a9cf78efc242" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251795 4335 cpu_manager.go:414] "containerMap: removing container" podUID="2a6e38bc-3e75-4392-90b2-a0472b37a443" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251800 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a6e38bc-3e75-4392-90b2-a0472b37a443" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251805 4335 cpu_manager.go:414] "containerMap: removing container" podUID="b117cee9-ce1b-421a-bd91-1d70a0483047" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251809 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b117cee9-ce1b-421a-bd91-1d70a0483047" containerName="pull" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251814 4335 cpu_manager.go:414] "containerMap: removing container" podUID="cca7fe9d-c55b-4597-843d-a9cf78efc242" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251818 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca7fe9d-c55b-4597-843d-a9cf78efc242" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251830 4335 cpu_manager.go:414] "containerMap: removing container" podUID="cd8714dd-e64c-440b-b110-6a2d087e5233" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251837 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd8714dd-e64c-440b-b110-6a2d087e5233" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251846 4335 cpu_manager.go:414] "containerMap: removing container" podUID="04670690-bbaa-4329-983b-9dcb81ada2c7" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251850 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04670690-bbaa-4329-983b-9dcb81ada2c7" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251857 4335 cpu_manager.go:414] "containerMap: removing container" podUID="b128ca38-8f49-4584-a0d0-f87fdc422db6" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251861 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b128ca38-8f49-4584-a0d0-f87fdc422db6" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251868 4335 cpu_manager.go:414] "containerMap: removing container" podUID="7e135608-c576-4f15-a0ae-6ebece176771" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251873 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e135608-c576-4f15-a0ae-6ebece176771" containerName="util" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251878 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f4ca5459-710c-48ee-9907-b9f1c0fb5fd7" containerName="extract" Oct 30 21:51:27.251843 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251883 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ca5459-710c-48ee-9907-b9f1c0fb5fd7" containerName="extract" Oct 30 21:51:27.254205 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251929 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="84db9e63-3579-46d7-a274-96e919a39275" containerName="extract" Oct 30 21:51:27.254205 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251940 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e135608-c576-4f15-a0ae-6ebece176771" containerName="extract" Oct 30 21:51:27.254205 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251946 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a6e38bc-3e75-4392-90b2-a0472b37a443" containerName="extract" Oct 30 21:51:27.254205 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251952 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f7a1ea3-fd94-4dfb-85c0-66d2a193fd03" containerName="extract" Oct 30 21:51:27.254205 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251957 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="140b47d4-0e7f-4e84-9d70-eaa209d01605" containerName="extract" Oct 30 21:51:27.254205 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251966 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="00124fd7-ac22-42ca-8fad-0047c9765e88" containerName="extract" Oct 30 21:51:27.254205 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251973 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="b117cee9-ce1b-421a-bd91-1d70a0483047" containerName="extract" Oct 30 21:51:27.254205 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251979 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ca5459-710c-48ee-9907-b9f1c0fb5fd7" containerName="extract" Oct 30 21:51:27.254205 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251984 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca7fe9d-c55b-4597-843d-a9cf78efc242" containerName="extract" Oct 30 21:51:27.254205 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251990 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="04670690-bbaa-4329-983b-9dcb81ada2c7" containerName="extract" Oct 30 21:51:27.254205 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251995 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd8714dd-e64c-440b-b110-6a2d087e5233" containerName="extract" Oct 30 21:51:27.254205 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.251999 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="b128ca38-8f49-4584-a0d0-f87fdc422db6" containerName="extract" Oct 30 21:51:27.255878 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.255854 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.308955 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.308931 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:51:27.309295 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.309234 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-ftnxp\"" Oct 30 21:51:27.309295 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.309242 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:51:27.309384 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.309318 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:51:27.309435 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.309395 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:51:27.309483 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.309441 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:51:27.327810 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.327788 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecdd3e01-9008-4557-852a-6f26931e1ddd-webhook-cert\") pod \"opendatahub-operator-controller-manager-68c9746c4b-dhlmg\" (UID: \"ecdd3e01-9008-4557-852a-6f26931e1ddd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.327860 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.327821 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcpq8\" (UniqueName: \"kubernetes.io/projected/ecdd3e01-9008-4557-852a-6f26931e1ddd-kube-api-access-xcpq8\") pod \"opendatahub-operator-controller-manager-68c9746c4b-dhlmg\" (UID: \"ecdd3e01-9008-4557-852a-6f26931e1ddd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.327860 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.327847 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ecdd3e01-9008-4557-852a-6f26931e1ddd-metrics-cert\") pod \"opendatahub-operator-controller-manager-68c9746c4b-dhlmg\" (UID: \"ecdd3e01-9008-4557-852a-6f26931e1ddd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.327919 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.327904 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ecdd3e01-9008-4557-852a-6f26931e1ddd-manifests\") pod \"opendatahub-operator-controller-manager-68c9746c4b-dhlmg\" (UID: \"ecdd3e01-9008-4557-852a-6f26931e1ddd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.327960 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.327927 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ecdd3e01-9008-4557-852a-6f26931e1ddd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68c9746c4b-dhlmg\" (UID: \"ecdd3e01-9008-4557-852a-6f26931e1ddd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.327995 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.327978 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecdd3e01-9008-4557-852a-6f26931e1ddd-tmp\") pod \"opendatahub-operator-controller-manager-68c9746c4b-dhlmg\" (UID: \"ecdd3e01-9008-4557-852a-6f26931e1ddd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.368331 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.368308 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg"] Oct 30 21:51:27.429054 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.429033 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcpq8\" (UniqueName: \"kubernetes.io/projected/ecdd3e01-9008-4557-852a-6f26931e1ddd-kube-api-access-xcpq8\") pod \"opendatahub-operator-controller-manager-68c9746c4b-dhlmg\" (UID: \"ecdd3e01-9008-4557-852a-6f26931e1ddd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.429164 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.429078 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ecdd3e01-9008-4557-852a-6f26931e1ddd-metrics-cert\") pod \"opendatahub-operator-controller-manager-68c9746c4b-dhlmg\" (UID: \"ecdd3e01-9008-4557-852a-6f26931e1ddd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.429164 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.429105 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ecdd3e01-9008-4557-852a-6f26931e1ddd-manifests\") pod \"opendatahub-operator-controller-manager-68c9746c4b-dhlmg\" (UID: \"ecdd3e01-9008-4557-852a-6f26931e1ddd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.429164 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.429129 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ecdd3e01-9008-4557-852a-6f26931e1ddd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68c9746c4b-dhlmg\" (UID: \"ecdd3e01-9008-4557-852a-6f26931e1ddd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.429164 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.429159 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecdd3e01-9008-4557-852a-6f26931e1ddd-tmp\") pod \"opendatahub-operator-controller-manager-68c9746c4b-dhlmg\" (UID: \"ecdd3e01-9008-4557-852a-6f26931e1ddd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.429373 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.429221 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecdd3e01-9008-4557-852a-6f26931e1ddd-webhook-cert\") pod \"opendatahub-operator-controller-manager-68c9746c4b-dhlmg\" (UID: \"ecdd3e01-9008-4557-852a-6f26931e1ddd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.429549 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.429519 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ecdd3e01-9008-4557-852a-6f26931e1ddd-manifests\") pod \"opendatahub-operator-controller-manager-68c9746c4b-dhlmg\" (UID: \"ecdd3e01-9008-4557-852a-6f26931e1ddd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.431560 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.431531 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecdd3e01-9008-4557-852a-6f26931e1ddd-tmp\") pod \"opendatahub-operator-controller-manager-68c9746c4b-dhlmg\" (UID: \"ecdd3e01-9008-4557-852a-6f26931e1ddd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.431777 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.431757 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecdd3e01-9008-4557-852a-6f26931e1ddd-webhook-cert\") pod \"opendatahub-operator-controller-manager-68c9746c4b-dhlmg\" (UID: \"ecdd3e01-9008-4557-852a-6f26931e1ddd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.431840 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.431783 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ecdd3e01-9008-4557-852a-6f26931e1ddd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68c9746c4b-dhlmg\" (UID: \"ecdd3e01-9008-4557-852a-6f26931e1ddd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.431947 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.431932 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ecdd3e01-9008-4557-852a-6f26931e1ddd-metrics-cert\") pod \"opendatahub-operator-controller-manager-68c9746c4b-dhlmg\" (UID: \"ecdd3e01-9008-4557-852a-6f26931e1ddd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.505777 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.505723 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcpq8\" (UniqueName: \"kubernetes.io/projected/ecdd3e01-9008-4557-852a-6f26931e1ddd-kube-api-access-xcpq8\") pod \"opendatahub-operator-controller-manager-68c9746c4b-dhlmg\" (UID: \"ecdd3e01-9008-4557-852a-6f26931e1ddd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:27.568273 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:27.568252 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:28.109260 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:28.109235 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg"] Oct 30 21:51:28.110427 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:51:28.110397 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecdd3e01_9008_4557_852a_6f26931e1ddd.slice/crio-fb7b9c93b9d7a3efe463f5d64f20d50c04604acdfdd7b996b6a540ed1e1999c6 WatchSource:0}: Error finding container fb7b9c93b9d7a3efe463f5d64f20d50c04604acdfdd7b996b6a540ed1e1999c6: Status 404 returned error can't find the container with id fb7b9c93b9d7a3efe463f5d64f20d50c04604acdfdd7b996b6a540ed1e1999c6 Oct 30 21:51:28.289705 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:28.289682 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" event={"ID":"ecdd3e01-9008-4557-852a-6f26931e1ddd","Type":"ContainerStarted","Data":"fb7b9c93b9d7a3efe463f5d64f20d50c04604acdfdd7b996b6a540ed1e1999c6"} Oct 30 21:51:30.141300 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:30.141266 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4"] Oct 30 21:51:30.144273 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:30.144253 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4" Oct 30 21:51:30.198040 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:30.198013 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 21:51:30.198175 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:30.198059 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-5vpsw\"" Oct 30 21:51:30.247906 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:30.247873 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:51:30.249821 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:30.249798 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4"] Oct 30 21:51:30.252218 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:30.252197 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebc469de-a12c-432f-b780-bfbca591adaf-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4\" (UID: \"ebc469de-a12c-432f-b780-bfbca591adaf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4" Oct 30 21:51:30.252338 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:30.252322 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6zzr\" (UniqueName: \"kubernetes.io/projected/ebc469de-a12c-432f-b780-bfbca591adaf-kube-api-access-l6zzr\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4\" (UID: \"ebc469de-a12c-432f-b780-bfbca591adaf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4" Oct 30 21:51:30.352869 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:30.352841 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebc469de-a12c-432f-b780-bfbca591adaf-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4\" (UID: \"ebc469de-a12c-432f-b780-bfbca591adaf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4" Oct 30 21:51:30.353038 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:30.352929 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6zzr\" (UniqueName: \"kubernetes.io/projected/ebc469de-a12c-432f-b780-bfbca591adaf-kube-api-access-l6zzr\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4\" (UID: \"ebc469de-a12c-432f-b780-bfbca591adaf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4" Oct 30 21:51:30.353194 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:30.353174 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebc469de-a12c-432f-b780-bfbca591adaf-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4\" (UID: \"ebc469de-a12c-432f-b780-bfbca591adaf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4" Oct 30 21:51:30.414309 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:30.414254 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6zzr\" (UniqueName: \"kubernetes.io/projected/ebc469de-a12c-432f-b780-bfbca591adaf-kube-api-access-l6zzr\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4\" (UID: \"ebc469de-a12c-432f-b780-bfbca591adaf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4" Oct 30 21:51:30.457238 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:30.457213 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4" Oct 30 21:51:30.935784 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:30.935564 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4"] Oct 30 21:51:30.939322 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:51:30.939282 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc469de_a12c_432f_b780_bfbca591adaf.slice/crio-745fbe6e779d324e6e1e6968aca5b39351def85e6f02ff657ca4f745307131d8 WatchSource:0}: Error finding container 745fbe6e779d324e6e1e6968aca5b39351def85e6f02ff657ca4f745307131d8: Status 404 returned error can't find the container with id 745fbe6e779d324e6e1e6968aca5b39351def85e6f02ff657ca4f745307131d8 Oct 30 21:51:31.309209 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:31.309174 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4" event={"ID":"ebc469de-a12c-432f-b780-bfbca591adaf","Type":"ContainerStarted","Data":"745fbe6e779d324e6e1e6968aca5b39351def85e6f02ff657ca4f745307131d8"} Oct 30 21:51:32.316020 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:32.315987 4335 generic.go:356] "Generic (PLEG): container finished" podID="ecdd3e01-9008-4557-852a-6f26931e1ddd" containerID="cee17965cff4462d8350976c0842852fcd9ec297d25cd314a9a23cd953972485" exitCode=0 Oct 30 21:51:32.316440 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:32.316076 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" event={"ID":"ecdd3e01-9008-4557-852a-6f26931e1ddd","Type":"ContainerDied","Data":"cee17965cff4462d8350976c0842852fcd9ec297d25cd314a9a23cd953972485"} Oct 30 21:51:33.055496 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.055466 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh"] Oct 30 21:51:33.057867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.057844 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh" Oct 30 21:51:33.073163 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.073141 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh\" (UID: \"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh" Oct 30 21:51:33.073266 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.073176 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh\" (UID: \"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh" Oct 30 21:51:33.073266 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.073205 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scrhv\" (UniqueName: \"kubernetes.io/projected/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-kube-api-access-scrhv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh\" (UID: \"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh" Oct 30 21:51:33.115804 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.115781 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-s5g56\"" Oct 30 21:51:33.120999 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.120977 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 21:51:33.121130 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.121016 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 21:51:33.144928 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.144902 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh"] Oct 30 21:51:33.174191 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.174166 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh\" (UID: \"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh" Oct 30 21:51:33.174292 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.174217 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh\" (UID: \"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh" Oct 30 21:51:33.174292 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.174250 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scrhv\" (UniqueName: \"kubernetes.io/projected/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-kube-api-access-scrhv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh\" (UID: \"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh" Oct 30 21:51:33.174541 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.174522 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh\" (UID: \"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh" Oct 30 21:51:33.174632 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.174591 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh\" (UID: \"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh" Oct 30 21:51:33.240785 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.240760 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scrhv\" (UniqueName: \"kubernetes.io/projected/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-kube-api-access-scrhv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh\" (UID: \"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh" Oct 30 21:51:33.323550 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.323481 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" event={"ID":"ecdd3e01-9008-4557-852a-6f26931e1ddd","Type":"ContainerStarted","Data":"12a00887e50829807876fc9bb06ab88d6e0d0a5f95f326ea9163b886fc61c9d8"} Oct 30 21:51:33.323958 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.323677 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:33.372140 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.372118 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh" Oct 30 21:51:33.469833 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.469743 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" podStartSLOduration=1.712600108 podStartE2EDuration="6.469727924s" totalImagesPullingTime="4.303731088s" totalInitContainerRuntime="453.396728ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:33.467054428 +0000 UTC m=+426.567486173" watchObservedRunningTime="2026-10-30 21:51:33.469727924 +0000 UTC m=+426.570159697" Oct 30 21:51:33.749771 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:33.749752 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh"] Oct 30 21:51:33.751160 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:51:33.751139 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10dad4a4_d48f_46c9_bdd2_2e28cd72c66c.slice/crio-6abd2076caa6f79e78f1ef2245f4cdbcad80c42b1e8530de7af6e32f0def2c05 WatchSource:0}: Error finding container 6abd2076caa6f79e78f1ef2245f4cdbcad80c42b1e8530de7af6e32f0def2c05: Status 404 returned error can't find the container with id 6abd2076caa6f79e78f1ef2245f4cdbcad80c42b1e8530de7af6e32f0def2c05 Oct 30 21:51:34.329962 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:34.329913 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4" event={"ID":"ebc469de-a12c-432f-b780-bfbca591adaf","Type":"ContainerStarted","Data":"c150a2e1b1a478fd4eb7e7021228c9ab300f93987d04cd60b5d3472365fbbff8"} Oct 30 21:51:34.331670 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:34.331641 4335 generic.go:356] "Generic (PLEG): container finished" podID="10dad4a4-d48f-46c9-bdd2-2e28cd72c66c" containerID="79649d43e7232350cdff75c44ce3957b0005c521cf2cf46885a0adc9a3668141" exitCode=0 Oct 30 21:51:34.331783 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:34.331714 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh" event={"ID":"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c","Type":"ContainerDied","Data":"79649d43e7232350cdff75c44ce3957b0005c521cf2cf46885a0adc9a3668141"} Oct 30 21:51:34.331783 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:34.331763 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh" event={"ID":"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c","Type":"ContainerStarted","Data":"6abd2076caa6f79e78f1ef2245f4cdbcad80c42b1e8530de7af6e32f0def2c05"} Oct 30 21:51:34.453029 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:34.452980 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9jb4" podStartSLOduration=1.6887020430000002 podStartE2EDuration="4.45296719s" totalImagesPullingTime="2.764265147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:34.451802784 +0000 UTC m=+427.552234535" watchObservedRunningTime="2026-10-30 21:51:34.45296719 +0000 UTC m=+427.553398977" Oct 30 21:51:35.338562 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:35.338528 4335 generic.go:356] "Generic (PLEG): container finished" podID="10dad4a4-d48f-46c9-bdd2-2e28cd72c66c" containerID="045eac305fca2ebb13202aef74145c2cd96cdae0cedbd31129f80a898a99a5b2" exitCode=0 Oct 30 21:51:35.338962 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:35.338643 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh" event={"ID":"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c","Type":"ContainerDied","Data":"045eac305fca2ebb13202aef74145c2cd96cdae0cedbd31129f80a898a99a5b2"} Oct 30 21:51:36.345422 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:36.345375 4335 generic.go:356] "Generic (PLEG): container finished" podID="10dad4a4-d48f-46c9-bdd2-2e28cd72c66c" containerID="934f8a8e725038288464b75430563f9451c5fe7d5cadbf965d2b822b6fd7edcc" exitCode=0 Oct 30 21:51:36.345800 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:36.345460 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh" event={"ID":"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c","Type":"ContainerDied","Data":"934f8a8e725038288464b75430563f9451c5fe7d5cadbf965d2b822b6fd7edcc"} Oct 30 21:51:37.474060 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:37.474037 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh" Oct 30 21:51:37.506156 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:37.506135 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-bundle\") pod \"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c\" (UID: \"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c\") " Oct 30 21:51:37.506262 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:37.506179 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-kube-api-access-scrhv\" (UniqueName: \"kubernetes.io/projected/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-kube-api-access-scrhv\") pod \"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c\" (UID: \"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c\") " Oct 30 21:51:37.506262 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:37.506204 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-util\" (UniqueName: \"kubernetes.io/empty-dir/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-util\") pod \"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c\" (UID: \"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c\") " Oct 30 21:51:37.506691 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:37.506663 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-bundle" pod "10dad4a4-d48f-46c9-bdd2-2e28cd72c66c" (UID: "10dad4a4-d48f-46c9-bdd2-2e28cd72c66c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:37.508266 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:37.508242 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-kube-api-access-scrhv" pod "10dad4a4-d48f-46c9-bdd2-2e28cd72c66c" (UID: "10dad4a4-d48f-46c9-bdd2-2e28cd72c66c"). InnerVolumeSpecName "kube-api-access-scrhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:37.513016 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:37.512992 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-util" pod "10dad4a4-d48f-46c9-bdd2-2e28cd72c66c" (UID: "10dad4a4-d48f-46c9-bdd2-2e28cd72c66c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:37.607282 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:37.607228 4335 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-bundle\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:37.607282 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:37.607259 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scrhv\" (UniqueName: \"kubernetes.io/projected/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-kube-api-access-scrhv\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:37.607282 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:37.607274 4335 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10dad4a4-d48f-46c9-bdd2-2e28cd72c66c-util\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:38.355992 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:38.355958 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh" event={"ID":"10dad4a4-d48f-46c9-bdd2-2e28cd72c66c","Type":"ContainerDied","Data":"6abd2076caa6f79e78f1ef2245f4cdbcad80c42b1e8530de7af6e32f0def2c05"} Oct 30 21:51:38.356157 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:38.356002 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6abd2076caa6f79e78f1ef2245f4cdbcad80c42b1e8530de7af6e32f0def2c05" Oct 30 21:51:38.356157 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:38.355999 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q8zmh" Oct 30 21:51:42.702776 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.702745 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rzj"] Oct 30 21:51:42.703159 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.703008 4335 cpu_manager.go:414] "containerMap: removing container" podUID="10dad4a4-d48f-46c9-bdd2-2e28cd72c66c" containerName="pull" Oct 30 21:51:42.703159 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.703019 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10dad4a4-d48f-46c9-bdd2-2e28cd72c66c" containerName="pull" Oct 30 21:51:42.703159 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.703037 4335 cpu_manager.go:414] "containerMap: removing container" podUID="10dad4a4-d48f-46c9-bdd2-2e28cd72c66c" containerName="util" Oct 30 21:51:42.703159 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.703042 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10dad4a4-d48f-46c9-bdd2-2e28cd72c66c" containerName="util" Oct 30 21:51:42.703159 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.703050 4335 cpu_manager.go:414] "containerMap: removing container" podUID="10dad4a4-d48f-46c9-bdd2-2e28cd72c66c" containerName="extract" Oct 30 21:51:42.703159 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.703055 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10dad4a4-d48f-46c9-bdd2-2e28cd72c66c" containerName="extract" Oct 30 21:51:42.703159 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.703106 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="10dad4a4-d48f-46c9-bdd2-2e28cd72c66c" containerName="extract" Oct 30 21:51:42.705877 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.705859 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rzj" Oct 30 21:51:42.744512 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.744488 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79dll\" (UniqueName: \"kubernetes.io/projected/73ddab2a-f425-4f3a-8a7e-ba0f12b5154d-kube-api-access-79dll\") pod \"cert-manager-cainjector-5c57bbd4fc-t5rzj\" (UID: \"73ddab2a-f425-4f3a-8a7e-ba0f12b5154d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rzj" Oct 30 21:51:42.744658 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.744518 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73ddab2a-f425-4f3a-8a7e-ba0f12b5154d-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-t5rzj\" (UID: \"73ddab2a-f425-4f3a-8a7e-ba0f12b5154d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rzj" Oct 30 21:51:42.757834 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.757810 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 21:51:42.807227 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.807205 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qb844\"" Oct 30 21:51:42.807338 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.807227 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 21:51:42.809483 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.809467 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rzj"] Oct 30 21:51:42.845531 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.845511 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79dll\" (UniqueName: \"kubernetes.io/projected/73ddab2a-f425-4f3a-8a7e-ba0f12b5154d-kube-api-access-79dll\") pod \"cert-manager-cainjector-5c57bbd4fc-t5rzj\" (UID: \"73ddab2a-f425-4f3a-8a7e-ba0f12b5154d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rzj" Oct 30 21:51:42.845661 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.845538 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73ddab2a-f425-4f3a-8a7e-ba0f12b5154d-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-t5rzj\" (UID: \"73ddab2a-f425-4f3a-8a7e-ba0f12b5154d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rzj" Oct 30 21:51:42.906726 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.906702 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73ddab2a-f425-4f3a-8a7e-ba0f12b5154d-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-t5rzj\" (UID: \"73ddab2a-f425-4f3a-8a7e-ba0f12b5154d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rzj" Oct 30 21:51:42.906794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:42.906753 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79dll\" (UniqueName: \"kubernetes.io/projected/73ddab2a-f425-4f3a-8a7e-ba0f12b5154d-kube-api-access-79dll\") pod \"cert-manager-cainjector-5c57bbd4fc-t5rzj\" (UID: \"73ddab2a-f425-4f3a-8a7e-ba0f12b5154d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rzj" Oct 30 21:51:43.025647 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:43.025575 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rzj" Oct 30 21:51:43.407847 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:43.407828 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rzj"] Oct 30 21:51:43.410169 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:51:43.410146 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73ddab2a_f425_4f3a_8a7e_ba0f12b5154d.slice/crio-f248d4d7653e0aa3e53c523b1e862567cb5a5cc0242ffc0026fc3853880f2e70 WatchSource:0}: Error finding container f248d4d7653e0aa3e53c523b1e862567cb5a5cc0242ffc0026fc3853880f2e70: Status 404 returned error can't find the container with id f248d4d7653e0aa3e53c523b1e862567cb5a5cc0242ffc0026fc3853880f2e70 Oct 30 21:51:44.334209 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:44.334184 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68c9746c4b-dhlmg" Oct 30 21:51:44.385952 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:44.385926 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rzj" event={"ID":"73ddab2a-f425-4f3a-8a7e-ba0f12b5154d","Type":"ContainerStarted","Data":"f248d4d7653e0aa3e53c523b1e862567cb5a5cc0242ffc0026fc3853880f2e70"} Oct 30 21:51:47.403719 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:47.403681 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rzj" event={"ID":"73ddab2a-f425-4f3a-8a7e-ba0f12b5154d","Type":"ContainerStarted","Data":"325b961632e581ce128cfcea2dd94789230d7b502e6bd21ea62507cb0d6e03c0"} Oct 30 21:51:47.523135 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:47.523093 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t5rzj" podStartSLOduration=1.9481152609999999 podStartE2EDuration="5.523078297s" totalImagesPullingTime="3.574963036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:47.522062897 +0000 UTC m=+440.622494629" watchObservedRunningTime="2026-10-30 21:51:47.523078297 +0000 UTC m=+440.623510029" Oct 30 21:51:50.383116 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:50.383085 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-dlqz5"] Oct 30 21:51:50.386514 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:50.386495 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dlqz5" Oct 30 21:51:50.488000 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:50.487972 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:51:50.488104 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:50.488027 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 21:51:50.488250 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:50.488235 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:51:50.488282 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:50.488236 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-h8bbh\"" Oct 30 21:51:50.490270 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:50.490242 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-dlqz5"] Oct 30 21:51:50.501424 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:50.501400 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67855\" (UniqueName: \"kubernetes.io/projected/7cdc1aa2-31c1-4e01-9ad1-58baa27940fa-kube-api-access-67855\") pod \"dns-operator-controller-manager-8557cbdb84-dlqz5\" (UID: \"7cdc1aa2-31c1-4e01-9ad1-58baa27940fa\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dlqz5" Oct 30 21:51:50.602255 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:50.602228 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67855\" (UniqueName: \"kubernetes.io/projected/7cdc1aa2-31c1-4e01-9ad1-58baa27940fa-kube-api-access-67855\") pod \"dns-operator-controller-manager-8557cbdb84-dlqz5\" (UID: \"7cdc1aa2-31c1-4e01-9ad1-58baa27940fa\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dlqz5" Oct 30 21:51:50.669017 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:50.668967 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67855\" (UniqueName: \"kubernetes.io/projected/7cdc1aa2-31c1-4e01-9ad1-58baa27940fa-kube-api-access-67855\") pod \"dns-operator-controller-manager-8557cbdb84-dlqz5\" (UID: \"7cdc1aa2-31c1-4e01-9ad1-58baa27940fa\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dlqz5" Oct 30 21:51:50.701773 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:50.701753 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dlqz5" Oct 30 21:51:51.072532 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:51.072509 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-dlqz5"] Oct 30 21:51:51.074249 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:51:51.074218 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cdc1aa2_31c1_4e01_9ad1_58baa27940fa.slice/crio-d89c1773117f542c57a854a5fc747a0591eacda92254caad744e1d4748ffdc58 WatchSource:0}: Error finding container d89c1773117f542c57a854a5fc747a0591eacda92254caad744e1d4748ffdc58: Status 404 returned error can't find the container with id d89c1773117f542c57a854a5fc747a0591eacda92254caad744e1d4748ffdc58 Oct 30 21:51:51.424147 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:51.424059 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dlqz5" event={"ID":"7cdc1aa2-31c1-4e01-9ad1-58baa27940fa","Type":"ContainerStarted","Data":"d89c1773117f542c57a854a5fc747a0591eacda92254caad744e1d4748ffdc58"} Oct 30 21:51:55.318452 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:51:55.318398 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:51:55.445909 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:55.445877 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dlqz5" event={"ID":"7cdc1aa2-31c1-4e01-9ad1-58baa27940fa","Type":"ContainerStarted","Data":"ee9845342024161c512ea4a28d609b189129911cfa03eb07c6351f810e09dd47"} Oct 30 21:51:55.446093 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:55.445966 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dlqz5" Oct 30 21:51:55.566109 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:55.566068 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dlqz5" podStartSLOduration=2.01316805 podStartE2EDuration="5.566053589s" totalImagesPullingTime="3.552885539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:55.564478934 +0000 UTC m=+448.664910667" watchObservedRunningTime="2026-10-30 21:51:55.566053589 +0000 UTC m=+448.666485326" Oct 30 21:51:55.616049 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:55.615995 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4"] Oct 30 21:51:55.619358 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:55.619343 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4" Oct 30 21:51:55.672150 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:55.672129 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mfkcm\"" Oct 30 21:51:55.729344 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:55.729318 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4"] Oct 30 21:51:55.743095 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:55.743068 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae7a6f4e-9d12-423e-b926-1e128349bc28-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-8bgj4\" (UID: \"ae7a6f4e-9d12-423e-b926-1e128349bc28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4" Oct 30 21:51:55.743214 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:55.743105 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mzfx\" (UniqueName: \"kubernetes.io/projected/ae7a6f4e-9d12-423e-b926-1e128349bc28-kube-api-access-8mzfx\") pod \"kuadrant-operator-controller-manager-cf8469d7b-8bgj4\" (UID: \"ae7a6f4e-9d12-423e-b926-1e128349bc28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4" Oct 30 21:51:55.843873 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:55.843843 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mzfx\" (UniqueName: \"kubernetes.io/projected/ae7a6f4e-9d12-423e-b926-1e128349bc28-kube-api-access-8mzfx\") pod \"kuadrant-operator-controller-manager-cf8469d7b-8bgj4\" (UID: \"ae7a6f4e-9d12-423e-b926-1e128349bc28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4" Oct 30 21:51:55.843988 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:55.843902 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae7a6f4e-9d12-423e-b926-1e128349bc28-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-8bgj4\" (UID: \"ae7a6f4e-9d12-423e-b926-1e128349bc28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4" Oct 30 21:51:55.844213 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:55.844198 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae7a6f4e-9d12-423e-b926-1e128349bc28-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-8bgj4\" (UID: \"ae7a6f4e-9d12-423e-b926-1e128349bc28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4" Oct 30 21:51:55.912255 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:55.912190 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mzfx\" (UniqueName: \"kubernetes.io/projected/ae7a6f4e-9d12-423e-b926-1e128349bc28-kube-api-access-8mzfx\") pod \"kuadrant-operator-controller-manager-cf8469d7b-8bgj4\" (UID: \"ae7a6f4e-9d12-423e-b926-1e128349bc28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4" Oct 30 21:51:55.931028 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:55.931005 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4" Oct 30 21:51:56.377426 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:56.377404 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4"] Oct 30 21:51:56.379767 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:51:56.379741 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7a6f4e_9d12_423e_b926_1e128349bc28.slice/crio-f3a989b3a5da59ff7632d7d61cdf74069fdadf06dd18d1088475f915f3509cb5 WatchSource:0}: Error finding container f3a989b3a5da59ff7632d7d61cdf74069fdadf06dd18d1088475f915f3509cb5: Status 404 returned error can't find the container with id f3a989b3a5da59ff7632d7d61cdf74069fdadf06dd18d1088475f915f3509cb5 Oct 30 21:51:56.452176 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:56.452147 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4" event={"ID":"ae7a6f4e-9d12-423e-b926-1e128349bc28","Type":"ContainerStarted","Data":"f3a989b3a5da59ff7632d7d61cdf74069fdadf06dd18d1088475f915f3509cb5"} Oct 30 21:51:57.982113 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:57.982083 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hwplz"] Oct 30 21:51:57.986663 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:57.986643 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hwplz" Oct 30 21:51:58.045303 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:58.045283 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xx58b\"" Oct 30 21:51:58.059976 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:58.059948 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqfwd\" (UniqueName: \"kubernetes.io/projected/8981dd9f-943f-4c22-88da-fb4395ff652b-kube-api-access-gqfwd\") pod \"authorino-operator-7b7bb4bc77-hwplz\" (UID: \"8981dd9f-943f-4c22-88da-fb4395ff652b\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hwplz" Oct 30 21:51:58.085323 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:58.085301 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hwplz"] Oct 30 21:51:58.161028 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:58.160998 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqfwd\" (UniqueName: \"kubernetes.io/projected/8981dd9f-943f-4c22-88da-fb4395ff652b-kube-api-access-gqfwd\") pod \"authorino-operator-7b7bb4bc77-hwplz\" (UID: \"8981dd9f-943f-4c22-88da-fb4395ff652b\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hwplz" Oct 30 21:51:58.221810 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:58.221789 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqfwd\" (UniqueName: \"kubernetes.io/projected/8981dd9f-943f-4c22-88da-fb4395ff652b-kube-api-access-gqfwd\") pod \"authorino-operator-7b7bb4bc77-hwplz\" (UID: \"8981dd9f-943f-4c22-88da-fb4395ff652b\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hwplz" Oct 30 21:51:58.299737 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:58.299709 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hwplz" Oct 30 21:51:58.707956 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:58.707855 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hwplz"] Oct 30 21:51:58.710535 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:51:58.710502 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8981dd9f_943f_4c22_88da_fb4395ff652b.slice/crio-cfe81c9479a7a25dd1459de175f57d5c9441acc62240e31a74a6b46130c9c776 WatchSource:0}: Error finding container cfe81c9479a7a25dd1459de175f57d5c9441acc62240e31a74a6b46130c9c776: Status 404 returned error can't find the container with id cfe81c9479a7a25dd1459de175f57d5c9441acc62240e31a74a6b46130c9c776 Oct 30 21:51:59.471365 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:51:59.471326 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hwplz" event={"ID":"8981dd9f-943f-4c22-88da-fb4395ff652b","Type":"ContainerStarted","Data":"cfe81c9479a7a25dd1459de175f57d5c9441acc62240e31a74a6b46130c9c776"} Oct 30 21:52:00.539686 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:00.539656 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-shrdq"] Oct 30 21:52:00.549912 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:00.549801 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-shrdq" Oct 30 21:52:00.651244 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:00.651189 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-mls7w\"" Oct 30 21:52:00.654377 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:00.654351 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-shrdq"] Oct 30 21:52:00.679853 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:00.679820 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztlrq\" (UniqueName: \"kubernetes.io/projected/8d1c279d-f94f-4021-9bab-ed080adc78ca-kube-api-access-ztlrq\") pod \"limitador-operator-controller-manager-5778bb4449-shrdq\" (UID: \"8d1c279d-f94f-4021-9bab-ed080adc78ca\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-shrdq" Oct 30 21:52:00.781202 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:00.781170 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztlrq\" (UniqueName: \"kubernetes.io/projected/8d1c279d-f94f-4021-9bab-ed080adc78ca-kube-api-access-ztlrq\") pod \"limitador-operator-controller-manager-5778bb4449-shrdq\" (UID: \"8d1c279d-f94f-4021-9bab-ed080adc78ca\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-shrdq" Oct 30 21:52:00.842231 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:00.842174 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztlrq\" (UniqueName: \"kubernetes.io/projected/8d1c279d-f94f-4021-9bab-ed080adc78ca-kube-api-access-ztlrq\") pod \"limitador-operator-controller-manager-5778bb4449-shrdq\" (UID: \"8d1c279d-f94f-4021-9bab-ed080adc78ca\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-shrdq" Oct 30 21:52:00.870122 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:00.870088 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-shrdq" Oct 30 21:52:01.249509 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:01.249483 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-shrdq"] Oct 30 21:52:01.252229 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:52:01.252199 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1c279d_f94f_4021_9bab_ed080adc78ca.slice/crio-b6e4519708616c26f023f87ca85c4efa249df491332bcc89e8c442cb38ec5a2f WatchSource:0}: Error finding container b6e4519708616c26f023f87ca85c4efa249df491332bcc89e8c442cb38ec5a2f: Status 404 returned error can't find the container with id b6e4519708616c26f023f87ca85c4efa249df491332bcc89e8c442cb38ec5a2f Oct 30 21:52:01.484929 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:01.484654 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-shrdq" event={"ID":"8d1c279d-f94f-4021-9bab-ed080adc78ca","Type":"ContainerStarted","Data":"b6e4519708616c26f023f87ca85c4efa249df491332bcc89e8c442cb38ec5a2f"} Oct 30 21:52:01.486982 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:01.486856 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hwplz" event={"ID":"8981dd9f-943f-4c22-88da-fb4395ff652b","Type":"ContainerStarted","Data":"9fd99aeac6f553ade384e9fb0c55c55c82dad3a2bad54f620a5e0342d59b0091"} Oct 30 21:52:01.486982 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:01.486944 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hwplz" Oct 30 21:52:01.605359 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:01.605305 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hwplz" podStartSLOduration=2.167274387 podStartE2EDuration="4.605288927s" totalImagesPullingTime="2.43801454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:01.604298243 +0000 UTC m=+454.704729989" watchObservedRunningTime="2026-10-30 21:52:01.605288927 +0000 UTC m=+454.705720658" Oct 30 21:52:03.501096 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:03.501060 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4" event={"ID":"ae7a6f4e-9d12-423e-b926-1e128349bc28","Type":"ContainerStarted","Data":"062076f528e4840e88b1d6055b3338918f1f159f4e5798df2d94a0168d0c9c84"} Oct 30 21:52:03.501495 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:03.501337 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4" Oct 30 21:52:03.722583 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:03.722525 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4" podStartSLOduration=2.59324709 podStartE2EDuration="8.722507873s" totalImagesPullingTime="6.129260783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:03.719047032 +0000 UTC m=+456.819478773" watchObservedRunningTime="2026-10-30 21:52:03.722507873 +0000 UTC m=+456.822939608" Oct 30 21:52:04.508964 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:04.508925 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-shrdq" event={"ID":"8d1c279d-f94f-4021-9bab-ed080adc78ca","Type":"ContainerStarted","Data":"cef17ff0c283909e66d31765e99a73627ccabf60dda75f7fecbece516c1b4bd5"} Oct 30 21:52:04.509359 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:04.509132 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-shrdq" Oct 30 21:52:04.626130 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:04.626076 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-shrdq" podStartSLOduration=2.037948704 podStartE2EDuration="4.626061733s" totalImagesPullingTime="2.588113029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:04.624724255 +0000 UTC m=+457.725155989" watchObservedRunningTime="2026-10-30 21:52:04.626061733 +0000 UTC m=+457.726493464" Oct 30 21:52:06.454697 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:06.454670 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dlqz5" Oct 30 21:52:12.494661 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:12.494597 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hwplz" Oct 30 21:52:14.510977 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:14.510946 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4" Oct 30 21:52:15.304416 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.304386 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx"] Oct 30 21:52:15.306552 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.306536 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx" Oct 30 21:52:15.392943 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.392915 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfztw\" (UniqueName: \"kubernetes.io/projected/6928f3c1-1469-4573-a58b-d1436438158c-kube-api-access-dfztw\") pod \"kuadrant-operator-controller-manager-678d87766f-vk7bx\" (UID: \"6928f3c1-1469-4573-a58b-d1436438158c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx" Oct 30 21:52:15.393054 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.392975 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6928f3c1-1469-4573-a58b-d1436438158c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vk7bx\" (UID: \"6928f3c1-1469-4573-a58b-d1436438158c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx" Oct 30 21:52:15.409358 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.409336 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx"] Oct 30 21:52:15.493773 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.493747 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6928f3c1-1469-4573-a58b-d1436438158c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vk7bx\" (UID: \"6928f3c1-1469-4573-a58b-d1436438158c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx" Oct 30 21:52:15.493883 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.493804 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfztw\" (UniqueName: \"kubernetes.io/projected/6928f3c1-1469-4573-a58b-d1436438158c-kube-api-access-dfztw\") pod \"kuadrant-operator-controller-manager-678d87766f-vk7bx\" (UID: \"6928f3c1-1469-4573-a58b-d1436438158c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx" Oct 30 21:52:15.494142 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.494124 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6928f3c1-1469-4573-a58b-d1436438158c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vk7bx\" (UID: \"6928f3c1-1469-4573-a58b-d1436438158c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx" Oct 30 21:52:15.522450 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.522424 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-shrdq" Oct 30 21:52:15.557917 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.557866 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfztw\" (UniqueName: \"kubernetes.io/projected/6928f3c1-1469-4573-a58b-d1436438158c-kube-api-access-dfztw\") pod \"kuadrant-operator-controller-manager-678d87766f-vk7bx\" (UID: \"6928f3c1-1469-4573-a58b-d1436438158c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx" Oct 30 21:52:15.618027 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.617999 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx" Oct 30 21:52:15.644896 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.644874 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf"] Oct 30 21:52:15.647242 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.647224 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.717909 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.717796 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf"] Oct 30 21:52:15.724042 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.724021 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-k92qm\"" Oct 30 21:52:15.802328 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.802301 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80256cf8-6ab3-405b-9e40-afdc5de94b5b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.802464 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.802344 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80256cf8-6ab3-405b-9e40-afdc5de94b5b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.802519 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.802493 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80256cf8-6ab3-405b-9e40-afdc5de94b5b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.802571 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.802553 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80256cf8-6ab3-405b-9e40-afdc5de94b5b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.802669 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.802650 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80256cf8-6ab3-405b-9e40-afdc5de94b5b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.802725 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.802684 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80256cf8-6ab3-405b-9e40-afdc5de94b5b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.802788 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.802763 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80256cf8-6ab3-405b-9e40-afdc5de94b5b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.802849 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.802829 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80256cf8-6ab3-405b-9e40-afdc5de94b5b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.802903 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.802866 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2csfc\" (UniqueName: \"kubernetes.io/projected/80256cf8-6ab3-405b-9e40-afdc5de94b5b-kube-api-access-2csfc\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.903648 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.903546 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80256cf8-6ab3-405b-9e40-afdc5de94b5b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.903648 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.903592 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80256cf8-6ab3-405b-9e40-afdc5de94b5b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.903870 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.903667 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80256cf8-6ab3-405b-9e40-afdc5de94b5b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.903870 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.903705 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80256cf8-6ab3-405b-9e40-afdc5de94b5b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.903870 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.903732 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80256cf8-6ab3-405b-9e40-afdc5de94b5b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.903870 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.903753 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80256cf8-6ab3-405b-9e40-afdc5de94b5b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.903870 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.903781 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80256cf8-6ab3-405b-9e40-afdc5de94b5b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.903870 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.903806 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80256cf8-6ab3-405b-9e40-afdc5de94b5b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.903870 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.903829 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2csfc\" (UniqueName: \"kubernetes.io/projected/80256cf8-6ab3-405b-9e40-afdc5de94b5b-kube-api-access-2csfc\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.904201 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.904103 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80256cf8-6ab3-405b-9e40-afdc5de94b5b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.904240 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.904205 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80256cf8-6ab3-405b-9e40-afdc5de94b5b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.904290 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.904267 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80256cf8-6ab3-405b-9e40-afdc5de94b5b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.904341 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.904279 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80256cf8-6ab3-405b-9e40-afdc5de94b5b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.904470 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.904453 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80256cf8-6ab3-405b-9e40-afdc5de94b5b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.906609 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.906585 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80256cf8-6ab3-405b-9e40-afdc5de94b5b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.906943 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.906923 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80256cf8-6ab3-405b-9e40-afdc5de94b5b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.965028 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.964996 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80256cf8-6ab3-405b-9e40-afdc5de94b5b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:15.966047 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:15.966028 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2csfc\" (UniqueName: \"kubernetes.io/projected/80256cf8-6ab3-405b-9e40-afdc5de94b5b-kube-api-access-2csfc\") pod \"data-science-gateway-data-science-gateway-class-549f9d654f2zkbf\" (UID: \"80256cf8-6ab3-405b-9e40-afdc5de94b5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:16.068867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:16.068847 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx"] Oct 30 21:52:16.070651 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:52:16.070629 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6928f3c1_1469_4573_a58b_d1436438158c.slice/crio-8318451354c601a4db9c68719a22bea539b40fef1926bb1fc22f31476c5a8b0e WatchSource:0}: Error finding container 8318451354c601a4db9c68719a22bea539b40fef1926bb1fc22f31476c5a8b0e: Status 404 returned error can't find the container with id 8318451354c601a4db9c68719a22bea539b40fef1926bb1fc22f31476c5a8b0e Oct 30 21:52:16.260226 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:16.260130 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:16.580396 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:16.580042 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx" event={"ID":"6928f3c1-1469-4573-a58b-d1436438158c","Type":"ContainerStarted","Data":"ff7a598a04a091a598df5dd1262c36204f93596440b269d01f7fbf81b75d8b6c"} Oct 30 21:52:16.580396 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:16.580142 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx" event={"ID":"6928f3c1-1469-4573-a58b-d1436438158c","Type":"ContainerStarted","Data":"8318451354c601a4db9c68719a22bea539b40fef1926bb1fc22f31476c5a8b0e"} Oct 30 21:52:16.581033 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:16.580932 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx" Oct 30 21:52:16.653809 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:16.653788 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf"] Oct 30 21:52:16.655697 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:52:16.655644 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80256cf8_6ab3_405b_9e40_afdc5de94b5b.slice/crio-56483e2e191cc10081c664fc5cbad3a63ff75daa0a77e020602ffc903ec8ef91 WatchSource:0}: Error finding container 56483e2e191cc10081c664fc5cbad3a63ff75daa0a77e020602ffc903ec8ef91: Status 404 returned error can't find the container with id 56483e2e191cc10081c664fc5cbad3a63ff75daa0a77e020602ffc903ec8ef91 Oct 30 21:52:16.658485 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:16.658456 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:52:16.658552 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:16.658533 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:52:16.658593 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:16.658574 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:52:16.711545 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:16.711490 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx" podStartSLOduration=1.7114717270000002 podStartE2EDuration="1.711471727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:16.709923949 +0000 UTC m=+469.810355681" watchObservedRunningTime="2026-10-30 21:52:16.711471727 +0000 UTC m=+469.811903460" Oct 30 21:52:17.587098 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:17.587068 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" event={"ID":"80256cf8-6ab3-405b-9e40-afdc5de94b5b","Type":"ContainerStarted","Data":"f75555b9d0575cd17e260407d2983382ba83d8e66e681b461a8a5678f11fd903"} Oct 30 21:52:17.587474 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:17.587116 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" event={"ID":"80256cf8-6ab3-405b-9e40-afdc5de94b5b","Type":"ContainerStarted","Data":"56483e2e191cc10081c664fc5cbad3a63ff75daa0a77e020602ffc903ec8ef91"} Oct 30 21:52:18.260399 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:18.260372 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:18.265547 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:18.265526 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:18.388254 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:18.388202 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" podStartSLOduration=3.388189123 podStartE2EDuration="3.388189123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:17.709158032 +0000 UTC m=+470.809589793" watchObservedRunningTime="2026-10-30 21:52:18.388189123 +0000 UTC m=+471.488620855" Oct 30 21:52:18.592558 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:18.592527 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:18.593289 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:18.593273 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654f2zkbf" Oct 30 21:52:25.319523 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:52:25.319489 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:52:28.594199 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:28.594172 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx" Oct 30 21:52:28.763924 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:28.763894 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4"] Oct 30 21:52:28.764098 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:28.764067 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4" podUID="ae7a6f4e-9d12-423e-b926-1e128349bc28" containerName="manager" containerID="cri-o://062076f528e4840e88b1d6055b3338918f1f159f4e5798df2d94a0168d0c9c84" gracePeriod=10 Oct 30 21:52:29.021954 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:29.021931 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4" Oct 30 21:52:29.107733 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:29.107699 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7a6f4e-9d12-423e-b926-1e128349bc28-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae7a6f4e-9d12-423e-b926-1e128349bc28-extensions-socket-volume\") pod \"ae7a6f4e-9d12-423e-b926-1e128349bc28\" (UID: \"ae7a6f4e-9d12-423e-b926-1e128349bc28\") " Oct 30 21:52:29.107891 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:29.107771 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7a6f4e-9d12-423e-b926-1e128349bc28-kube-api-access-8mzfx\" (UniqueName: \"kubernetes.io/projected/ae7a6f4e-9d12-423e-b926-1e128349bc28-kube-api-access-8mzfx\") pod \"ae7a6f4e-9d12-423e-b926-1e128349bc28\" (UID: \"ae7a6f4e-9d12-423e-b926-1e128349bc28\") " Oct 30 21:52:29.108117 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:29.108091 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7a6f4e-9d12-423e-b926-1e128349bc28-extensions-socket-volume" pod "ae7a6f4e-9d12-423e-b926-1e128349bc28" (UID: "ae7a6f4e-9d12-423e-b926-1e128349bc28"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:29.110073 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:29.110040 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7a6f4e-9d12-423e-b926-1e128349bc28-kube-api-access-8mzfx" pod "ae7a6f4e-9d12-423e-b926-1e128349bc28" (UID: "ae7a6f4e-9d12-423e-b926-1e128349bc28"). InnerVolumeSpecName "kube-api-access-8mzfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:29.208924 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:29.208865 4335 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae7a6f4e-9d12-423e-b926-1e128349bc28-extensions-socket-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:29.208924 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:29.208890 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mzfx\" (UniqueName: \"kubernetes.io/projected/ae7a6f4e-9d12-423e-b926-1e128349bc28-kube-api-access-8mzfx\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:29.656154 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:29.656116 4335 generic.go:356] "Generic (PLEG): container finished" podID="ae7a6f4e-9d12-423e-b926-1e128349bc28" containerID="062076f528e4840e88b1d6055b3338918f1f159f4e5798df2d94a0168d0c9c84" exitCode=0 Oct 30 21:52:29.656607 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:29.656164 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4" Oct 30 21:52:29.656607 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:29.656165 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4" event={"ID":"ae7a6f4e-9d12-423e-b926-1e128349bc28","Type":"ContainerDied","Data":"062076f528e4840e88b1d6055b3338918f1f159f4e5798df2d94a0168d0c9c84"} Oct 30 21:52:29.656607 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:29.656291 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4" event={"ID":"ae7a6f4e-9d12-423e-b926-1e128349bc28","Type":"ContainerDied","Data":"f3a989b3a5da59ff7632d7d61cdf74069fdadf06dd18d1088475f915f3509cb5"} Oct 30 21:52:29.656607 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:29.656315 4335 scope.go:122] "RemoveContainer" containerID="062076f528e4840e88b1d6055b3338918f1f159f4e5798df2d94a0168d0c9c84" Oct 30 21:52:29.665804 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:29.665789 4335 scope.go:122] "RemoveContainer" containerID="062076f528e4840e88b1d6055b3338918f1f159f4e5798df2d94a0168d0c9c84" Oct 30 21:52:29.666077 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:52:29.666040 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"062076f528e4840e88b1d6055b3338918f1f159f4e5798df2d94a0168d0c9c84\": container with ID starting with 062076f528e4840e88b1d6055b3338918f1f159f4e5798df2d94a0168d0c9c84 not found: ID does not exist" containerID="062076f528e4840e88b1d6055b3338918f1f159f4e5798df2d94a0168d0c9c84" Oct 30 21:52:29.666195 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:29.666091 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"062076f528e4840e88b1d6055b3338918f1f159f4e5798df2d94a0168d0c9c84"} err="failed to get container status \"062076f528e4840e88b1d6055b3338918f1f159f4e5798df2d94a0168d0c9c84\": rpc error: code = NotFound desc = could not find container \"062076f528e4840e88b1d6055b3338918f1f159f4e5798df2d94a0168d0c9c84\": container with ID starting with 062076f528e4840e88b1d6055b3338918f1f159f4e5798df2d94a0168d0c9c84 not found: ID does not exist" Oct 30 21:52:29.830132 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:29.830102 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4"] Oct 30 21:52:29.879470 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:29.879444 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8bgj4"] Oct 30 21:52:31.057183 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:31.057138 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae7a6f4e-9d12-423e-b926-1e128349bc28" path="/var/lib/kubelet/pods/ae7a6f4e-9d12-423e-b926-1e128349bc28/volumes" Oct 30 21:52:33.266896 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:33.266864 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9vj2n"] Oct 30 21:52:33.267277 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:33.267166 4335 cpu_manager.go:414] "containerMap: removing container" podUID="ae7a6f4e-9d12-423e-b926-1e128349bc28" containerName="manager" Oct 30 21:52:33.267277 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:33.267177 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7a6f4e-9d12-423e-b926-1e128349bc28" containerName="manager" Oct 30 21:52:33.267277 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:33.267236 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae7a6f4e-9d12-423e-b926-1e128349bc28" containerName="manager" Oct 30 21:52:33.270284 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:33.270267 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9vj2n" Oct 30 21:52:33.325195 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:33.325170 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 21:52:33.374430 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:33.374403 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kwpp9\"" Oct 30 21:52:33.376456 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:33.376437 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9vj2n"] Oct 30 21:52:33.440370 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:33.440342 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b448z\" (UniqueName: \"kubernetes.io/projected/908a7fb8-d307-400b-8cb2-73706f36526b-kube-api-access-b448z\") pod \"limitador-limitador-74dfcfb894-9vj2n\" (UID: \"908a7fb8-d307-400b-8cb2-73706f36526b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9vj2n" Oct 30 21:52:33.440469 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:33.440381 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/908a7fb8-d307-400b-8cb2-73706f36526b-config-file\") pod \"limitador-limitador-74dfcfb894-9vj2n\" (UID: \"908a7fb8-d307-400b-8cb2-73706f36526b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9vj2n" Oct 30 21:52:33.509629 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:33.509578 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9vj2n"] Oct 30 21:52:33.541659 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:33.541595 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b448z\" (UniqueName: \"kubernetes.io/projected/908a7fb8-d307-400b-8cb2-73706f36526b-kube-api-access-b448z\") pod \"limitador-limitador-74dfcfb894-9vj2n\" (UID: \"908a7fb8-d307-400b-8cb2-73706f36526b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9vj2n" Oct 30 21:52:33.541749 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:33.541655 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/908a7fb8-d307-400b-8cb2-73706f36526b-config-file\") pod \"limitador-limitador-74dfcfb894-9vj2n\" (UID: \"908a7fb8-d307-400b-8cb2-73706f36526b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9vj2n" Oct 30 21:52:33.542240 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:33.542216 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/908a7fb8-d307-400b-8cb2-73706f36526b-config-file\") pod \"limitador-limitador-74dfcfb894-9vj2n\" (UID: \"908a7fb8-d307-400b-8cb2-73706f36526b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9vj2n" Oct 30 21:52:33.602212 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:33.602189 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b448z\" (UniqueName: \"kubernetes.io/projected/908a7fb8-d307-400b-8cb2-73706f36526b-kube-api-access-b448z\") pod \"limitador-limitador-74dfcfb894-9vj2n\" (UID: \"908a7fb8-d307-400b-8cb2-73706f36526b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9vj2n" Oct 30 21:52:33.881768 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:33.881747 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9vj2n" Oct 30 21:52:34.258220 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:34.258197 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9vj2n"] Oct 30 21:52:34.260280 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:52:34.260252 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908a7fb8_d307_400b_8cb2_73706f36526b.slice/crio-261b46d536046e670b51716d379a29b8bb1671c54068f5886e51daba079d07ab WatchSource:0}: Error finding container 261b46d536046e670b51716d379a29b8bb1671c54068f5886e51daba079d07ab: Status 404 returned error can't find the container with id 261b46d536046e670b51716d379a29b8bb1671c54068f5886e51daba079d07ab Oct 30 21:52:34.684844 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:34.684809 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9vj2n" event={"ID":"908a7fb8-d307-400b-8cb2-73706f36526b","Type":"ContainerStarted","Data":"261b46d536046e670b51716d379a29b8bb1671c54068f5886e51daba079d07ab"} Oct 30 21:52:36.698800 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:36.698726 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9vj2n" event={"ID":"908a7fb8-d307-400b-8cb2-73706f36526b","Type":"ContainerStarted","Data":"77c8d9ac3de0a0d06a90560248aad0bc3be956f0486e4d34457fbe14a73e55eb"} Oct 30 21:52:36.699235 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:36.698831 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9vj2n" Oct 30 21:52:36.814437 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:36.814392 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-9vj2n" podStartSLOduration=1.694927514 podStartE2EDuration="3.81437841s" totalImagesPullingTime="2.119450896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:36.8133923 +0000 UTC m=+489.913824038" watchObservedRunningTime="2026-10-30 21:52:36.81437841 +0000 UTC m=+489.914810176" Oct 30 21:52:46.852899 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:46.852824 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 21:52:46.855207 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:46.855187 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-2gpxg" Oct 30 21:52:46.914276 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:46.914252 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 21:52:46.940648 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:46.940609 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6f28\" (UniqueName: \"kubernetes.io/projected/bc6e4759-93b2-49bd-ae8f-66a8ff627f02-kube-api-access-c6f28\") pod \"limitador-limitador-7c9b87d574-2gpxg\" (UID: \"bc6e4759-93b2-49bd-ae8f-66a8ff627f02\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2gpxg" Oct 30 21:52:46.940762 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:46.940693 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc6e4759-93b2-49bd-ae8f-66a8ff627f02-config-file\") pod \"limitador-limitador-7c9b87d574-2gpxg\" (UID: \"bc6e4759-93b2-49bd-ae8f-66a8ff627f02\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2gpxg" Oct 30 21:52:47.041725 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:47.041689 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6f28\" (UniqueName: \"kubernetes.io/projected/bc6e4759-93b2-49bd-ae8f-66a8ff627f02-kube-api-access-c6f28\") pod \"limitador-limitador-7c9b87d574-2gpxg\" (UID: \"bc6e4759-93b2-49bd-ae8f-66a8ff627f02\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2gpxg" Oct 30 21:52:47.041873 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:47.041755 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc6e4759-93b2-49bd-ae8f-66a8ff627f02-config-file\") pod \"limitador-limitador-7c9b87d574-2gpxg\" (UID: \"bc6e4759-93b2-49bd-ae8f-66a8ff627f02\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2gpxg" Oct 30 21:52:47.042416 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:47.042395 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc6e4759-93b2-49bd-ae8f-66a8ff627f02-config-file\") pod \"limitador-limitador-7c9b87d574-2gpxg\" (UID: \"bc6e4759-93b2-49bd-ae8f-66a8ff627f02\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2gpxg" Oct 30 21:52:47.114960 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:47.114901 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6f28\" (UniqueName: \"kubernetes.io/projected/bc6e4759-93b2-49bd-ae8f-66a8ff627f02-kube-api-access-c6f28\") pod \"limitador-limitador-7c9b87d574-2gpxg\" (UID: \"bc6e4759-93b2-49bd-ae8f-66a8ff627f02\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2gpxg" Oct 30 21:52:47.166872 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:47.166844 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-2gpxg" Oct 30 21:52:47.182136 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:47.182110 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 21:52:47.541313 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:47.541288 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 21:52:47.542176 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:52:47.542137 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6e4759_93b2_49bd_ae8f_66a8ff627f02.slice/crio-0e559d4b6f6b8ce44fb3a0e5040fcad3dae99e37f7ee037a4432876e56c66d42 WatchSource:0}: Error finding container 0e559d4b6f6b8ce44fb3a0e5040fcad3dae99e37f7ee037a4432876e56c66d42: Status 404 returned error can't find the container with id 0e559d4b6f6b8ce44fb3a0e5040fcad3dae99e37f7ee037a4432876e56c66d42 Oct 30 21:52:47.704972 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:47.704944 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9vj2n" Oct 30 21:52:47.754399 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:47.754365 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-2gpxg" event={"ID":"bc6e4759-93b2-49bd-ae8f-66a8ff627f02","Type":"ContainerStarted","Data":"759e00f3732425301dedd6ffa3015833bcf1b93fdc61e31488c417f951d1e79d"} Oct 30 21:52:47.754534 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:47.754415 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-2gpxg" event={"ID":"bc6e4759-93b2-49bd-ae8f-66a8ff627f02","Type":"ContainerStarted","Data":"0e559d4b6f6b8ce44fb3a0e5040fcad3dae99e37f7ee037a4432876e56c66d42"} Oct 30 21:52:47.754534 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:47.754501 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-2gpxg" Oct 30 21:52:47.938381 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:47.938330 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-2gpxg" podStartSLOduration=1.9383163639999998 podStartE2EDuration="1.938316364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:47.936310437 +0000 UTC m=+501.036742168" watchObservedRunningTime="2026-10-30 21:52:47.938316364 +0000 UTC m=+501.038748095" Oct 30 21:52:55.320539 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:52:55.320504 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:52:58.760156 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:58.760128 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-2gpxg" Oct 30 21:52:58.935775 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:58.935740 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9vj2n"] Oct 30 21:52:58.935966 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:58.935943 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-9vj2n" podUID="908a7fb8-d307-400b-8cb2-73706f36526b" containerName="limitador" containerID="cri-o://77c8d9ac3de0a0d06a90560248aad0bc3be956f0486e4d34457fbe14a73e55eb" gracePeriod=30 Oct 30 21:52:59.822794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:59.822766 4335 generic.go:356] "Generic (PLEG): container finished" podID="908a7fb8-d307-400b-8cb2-73706f36526b" containerID="77c8d9ac3de0a0d06a90560248aad0bc3be956f0486e4d34457fbe14a73e55eb" exitCode=0 Oct 30 21:52:59.823137 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:59.822814 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9vj2n" event={"ID":"908a7fb8-d307-400b-8cb2-73706f36526b","Type":"ContainerDied","Data":"77c8d9ac3de0a0d06a90560248aad0bc3be956f0486e4d34457fbe14a73e55eb"} Oct 30 21:52:59.876788 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:59.876766 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9vj2n" Oct 30 21:52:59.929631 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:59.929593 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908a7fb8-d307-400b-8cb2-73706f36526b-kube-api-access-b448z\" (UniqueName: \"kubernetes.io/projected/908a7fb8-d307-400b-8cb2-73706f36526b-kube-api-access-b448z\") pod \"908a7fb8-d307-400b-8cb2-73706f36526b\" (UID: \"908a7fb8-d307-400b-8cb2-73706f36526b\") " Oct 30 21:52:59.929724 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:59.929665 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/908a7fb8-d307-400b-8cb2-73706f36526b-config-file\" (UniqueName: \"kubernetes.io/configmap/908a7fb8-d307-400b-8cb2-73706f36526b-config-file\") pod \"908a7fb8-d307-400b-8cb2-73706f36526b\" (UID: \"908a7fb8-d307-400b-8cb2-73706f36526b\") " Oct 30 21:52:59.930000 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:59.929981 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908a7fb8-d307-400b-8cb2-73706f36526b-config-file" pod "908a7fb8-d307-400b-8cb2-73706f36526b" (UID: "908a7fb8-d307-400b-8cb2-73706f36526b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:59.931627 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:52:59.931592 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908a7fb8-d307-400b-8cb2-73706f36526b-kube-api-access-b448z" pod "908a7fb8-d307-400b-8cb2-73706f36526b" (UID: "908a7fb8-d307-400b-8cb2-73706f36526b"). InnerVolumeSpecName "kube-api-access-b448z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:00.030426 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:00.030358 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b448z\" (UniqueName: \"kubernetes.io/projected/908a7fb8-d307-400b-8cb2-73706f36526b-kube-api-access-b448z\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:00.030426 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:00.030394 4335 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/908a7fb8-d307-400b-8cb2-73706f36526b-config-file\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:00.829166 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:00.829141 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9vj2n" Oct 30 21:53:00.829545 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:00.829166 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9vj2n" event={"ID":"908a7fb8-d307-400b-8cb2-73706f36526b","Type":"ContainerDied","Data":"261b46d536046e670b51716d379a29b8bb1671c54068f5886e51daba079d07ab"} Oct 30 21:53:00.829545 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:00.829219 4335 scope.go:122] "RemoveContainer" containerID="77c8d9ac3de0a0d06a90560248aad0bc3be956f0486e4d34457fbe14a73e55eb" Oct 30 21:53:00.996756 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:00.996734 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9vj2n"] Oct 30 21:53:01.005309 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:01.005292 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9vj2n"] Oct 30 21:53:01.054874 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:01.054848 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="908a7fb8-d307-400b-8cb2-73706f36526b" path="/var/lib/kubelet/pods/908a7fb8-d307-400b-8cb2-73706f36526b/volumes" Oct 30 21:53:25.318992 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:53:25.318954 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:53:37.688997 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.688964 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-7m8xr"] Oct 30 21:53:37.689562 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.689433 4335 cpu_manager.go:414] "containerMap: removing container" podUID="908a7fb8-d307-400b-8cb2-73706f36526b" containerName="limitador" Oct 30 21:53:37.689562 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.689452 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908a7fb8-d307-400b-8cb2-73706f36526b" containerName="limitador" Oct 30 21:53:37.689718 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.689563 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="908a7fb8-d307-400b-8cb2-73706f36526b" containerName="limitador" Oct 30 21:53:37.692692 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.692659 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-7m8xr" Oct 30 21:53:37.796441 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.796405 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 21:53:37.796577 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.796446 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sr62x\"" Oct 30 21:53:37.796577 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.796457 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 21:53:37.796577 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.796492 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:53:37.796577 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.796507 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 21:53:37.796577 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.796508 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:53:37.811270 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.811247 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7k7j\" (UniqueName: \"kubernetes.io/projected/38a56e90-ae42-4011-9ae4-f13df5fd5254-kube-api-access-k7k7j\") pod \"maas-controller-579cf7c6d-7m8xr\" (UID: \"38a56e90-ae42-4011-9ae4-f13df5fd5254\") " pod="opendatahub/maas-controller-579cf7c6d-7m8xr" Oct 30 21:53:37.811395 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.811302 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a56e90-ae42-4011-9ae4-f13df5fd5254-metrics-certs\") pod \"maas-controller-579cf7c6d-7m8xr\" (UID: \"38a56e90-ae42-4011-9ae4-f13df5fd5254\") " pod="opendatahub/maas-controller-579cf7c6d-7m8xr" Oct 30 21:53:37.811466 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.811443 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38a56e90-ae42-4011-9ae4-f13df5fd5254-webhook-cert\") pod \"maas-controller-579cf7c6d-7m8xr\" (UID: \"38a56e90-ae42-4011-9ae4-f13df5fd5254\") " pod="opendatahub/maas-controller-579cf7c6d-7m8xr" Oct 30 21:53:37.814203 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.814184 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-7m8xr"] Oct 30 21:53:37.912258 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.912236 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a56e90-ae42-4011-9ae4-f13df5fd5254-metrics-certs\") pod \"maas-controller-579cf7c6d-7m8xr\" (UID: \"38a56e90-ae42-4011-9ae4-f13df5fd5254\") " pod="opendatahub/maas-controller-579cf7c6d-7m8xr" Oct 30 21:53:37.912359 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.912276 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38a56e90-ae42-4011-9ae4-f13df5fd5254-webhook-cert\") pod \"maas-controller-579cf7c6d-7m8xr\" (UID: \"38a56e90-ae42-4011-9ae4-f13df5fd5254\") " pod="opendatahub/maas-controller-579cf7c6d-7m8xr" Oct 30 21:53:37.912359 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.912303 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7k7j\" (UniqueName: \"kubernetes.io/projected/38a56e90-ae42-4011-9ae4-f13df5fd5254-kube-api-access-k7k7j\") pod \"maas-controller-579cf7c6d-7m8xr\" (UID: \"38a56e90-ae42-4011-9ae4-f13df5fd5254\") " pod="opendatahub/maas-controller-579cf7c6d-7m8xr" Oct 30 21:53:37.915494 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.915473 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a56e90-ae42-4011-9ae4-f13df5fd5254-metrics-certs\") pod \"maas-controller-579cf7c6d-7m8xr\" (UID: \"38a56e90-ae42-4011-9ae4-f13df5fd5254\") " pod="opendatahub/maas-controller-579cf7c6d-7m8xr" Oct 30 21:53:37.915737 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.915710 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38a56e90-ae42-4011-9ae4-f13df5fd5254-webhook-cert\") pod \"maas-controller-579cf7c6d-7m8xr\" (UID: \"38a56e90-ae42-4011-9ae4-f13df5fd5254\") " pod="opendatahub/maas-controller-579cf7c6d-7m8xr" Oct 30 21:53:37.984156 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:37.984096 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7k7j\" (UniqueName: \"kubernetes.io/projected/38a56e90-ae42-4011-9ae4-f13df5fd5254-kube-api-access-k7k7j\") pod \"maas-controller-579cf7c6d-7m8xr\" (UID: \"38a56e90-ae42-4011-9ae4-f13df5fd5254\") " pod="opendatahub/maas-controller-579cf7c6d-7m8xr" Oct 30 21:53:38.007101 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:38.007079 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-7m8xr" Oct 30 21:53:38.185954 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:38.185924 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bb56545f7-h6p47"] Oct 30 21:53:38.188140 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:38.188126 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bb56545f7-h6p47" Oct 30 21:53:38.260662 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:38.260536 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bb56545f7-h6p47"] Oct 30 21:53:38.315494 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:38.315459 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25fe090-0a01-4e56-8337-f1edaa4ac618-metrics-certs\") pod \"maas-controller-7bb56545f7-h6p47\" (UID: \"d25fe090-0a01-4e56-8337-f1edaa4ac618\") " pod="opendatahub/maas-controller-7bb56545f7-h6p47" Oct 30 21:53:38.315494 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:38.315494 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d25fe090-0a01-4e56-8337-f1edaa4ac618-webhook-cert\") pod \"maas-controller-7bb56545f7-h6p47\" (UID: \"d25fe090-0a01-4e56-8337-f1edaa4ac618\") " pod="opendatahub/maas-controller-7bb56545f7-h6p47" Oct 30 21:53:38.315774 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:38.315706 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9msh8\" (UniqueName: \"kubernetes.io/projected/d25fe090-0a01-4e56-8337-f1edaa4ac618-kube-api-access-9msh8\") pod \"maas-controller-7bb56545f7-h6p47\" (UID: \"d25fe090-0a01-4e56-8337-f1edaa4ac618\") " pod="opendatahub/maas-controller-7bb56545f7-h6p47" Oct 30 21:53:38.416442 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:38.416410 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9msh8\" (UniqueName: \"kubernetes.io/projected/d25fe090-0a01-4e56-8337-f1edaa4ac618-kube-api-access-9msh8\") pod \"maas-controller-7bb56545f7-h6p47\" (UID: \"d25fe090-0a01-4e56-8337-f1edaa4ac618\") " pod="opendatahub/maas-controller-7bb56545f7-h6p47" Oct 30 21:53:38.416442 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:38.416444 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25fe090-0a01-4e56-8337-f1edaa4ac618-metrics-certs\") pod \"maas-controller-7bb56545f7-h6p47\" (UID: \"d25fe090-0a01-4e56-8337-f1edaa4ac618\") " pod="opendatahub/maas-controller-7bb56545f7-h6p47" Oct 30 21:53:38.416708 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:38.416464 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d25fe090-0a01-4e56-8337-f1edaa4ac618-webhook-cert\") pod \"maas-controller-7bb56545f7-h6p47\" (UID: \"d25fe090-0a01-4e56-8337-f1edaa4ac618\") " pod="opendatahub/maas-controller-7bb56545f7-h6p47" Oct 30 21:53:38.418943 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:38.418910 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25fe090-0a01-4e56-8337-f1edaa4ac618-metrics-certs\") pod \"maas-controller-7bb56545f7-h6p47\" (UID: \"d25fe090-0a01-4e56-8337-f1edaa4ac618\") " pod="opendatahub/maas-controller-7bb56545f7-h6p47" Oct 30 21:53:38.418943 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:38.418941 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d25fe090-0a01-4e56-8337-f1edaa4ac618-webhook-cert\") pod \"maas-controller-7bb56545f7-h6p47\" (UID: \"d25fe090-0a01-4e56-8337-f1edaa4ac618\") " pod="opendatahub/maas-controller-7bb56545f7-h6p47" Oct 30 21:53:38.476929 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:38.476909 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-7m8xr"] Oct 30 21:53:38.478739 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:53:38.478718 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a56e90_ae42_4011_9ae4_f13df5fd5254.slice/crio-025a2c5c291d6b98e98eb70de525ae6cebf704de14548200911251256fec622f WatchSource:0}: Error finding container 025a2c5c291d6b98e98eb70de525ae6cebf704de14548200911251256fec622f: Status 404 returned error can't find the container with id 025a2c5c291d6b98e98eb70de525ae6cebf704de14548200911251256fec622f Oct 30 21:53:38.482030 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:38.482009 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9msh8\" (UniqueName: \"kubernetes.io/projected/d25fe090-0a01-4e56-8337-f1edaa4ac618-kube-api-access-9msh8\") pod \"maas-controller-7bb56545f7-h6p47\" (UID: \"d25fe090-0a01-4e56-8337-f1edaa4ac618\") " pod="opendatahub/maas-controller-7bb56545f7-h6p47" Oct 30 21:53:38.500007 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:38.499988 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bb56545f7-h6p47" Oct 30 21:53:38.892267 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:38.892242 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bb56545f7-h6p47"] Oct 30 21:53:38.894077 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:53:38.894054 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25fe090_0a01_4e56_8337_f1edaa4ac618.slice/crio-78bc19eb1cb491be01912940937342c2548cd5f9498e13557f90445e958b5ad4 WatchSource:0}: Error finding container 78bc19eb1cb491be01912940937342c2548cd5f9498e13557f90445e958b5ad4: Status 404 returned error can't find the container with id 78bc19eb1cb491be01912940937342c2548cd5f9498e13557f90445e958b5ad4 Oct 30 21:53:39.038668 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:39.038636 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bb56545f7-h6p47" event={"ID":"d25fe090-0a01-4e56-8337-f1edaa4ac618","Type":"ContainerStarted","Data":"78bc19eb1cb491be01912940937342c2548cd5f9498e13557f90445e958b5ad4"} Oct 30 21:53:39.039783 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:39.039760 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-7m8xr" event={"ID":"38a56e90-ae42-4011-9ae4-f13df5fd5254","Type":"ContainerStarted","Data":"025a2c5c291d6b98e98eb70de525ae6cebf704de14548200911251256fec622f"} Oct 30 21:53:42.065737 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:42.065708 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-7m8xr" event={"ID":"38a56e90-ae42-4011-9ae4-f13df5fd5254","Type":"ContainerStarted","Data":"beb94ab5922b64f4d97ec21f236bcd996dbc7b1400d8be302f385e65dee3a6f5"} Oct 30 21:53:42.066104 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:42.065873 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-7m8xr" Oct 30 21:53:42.216716 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:42.216651 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-7m8xr" podStartSLOduration=1.997096215 podStartE2EDuration="5.216635471s" totalImagesPullingTime="3.219539256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:42.214131991 +0000 UTC m=+555.314563723" watchObservedRunningTime="2026-10-30 21:53:42.216635471 +0000 UTC m=+555.317067200" Oct 30 21:53:43.072171 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:43.072127 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bb56545f7-h6p47" event={"ID":"d25fe090-0a01-4e56-8337-f1edaa4ac618","Type":"ContainerStarted","Data":"2f3ac74c472a212a0b63e23960099bfd4a54c52a83a0cc05a023b354dc20e0d3"} Oct 30 21:53:43.072749 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:43.072317 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bb56545f7-h6p47" Oct 30 21:53:43.222802 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:43.222747 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7bb56545f7-h6p47" podStartSLOduration=2.082718979 podStartE2EDuration="5.222731926s" totalImagesPullingTime="3.140012947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:43.221552585 +0000 UTC m=+556.321984317" watchObservedRunningTime="2026-10-30 21:53:43.222731926 +0000 UTC m=+556.323163658" Oct 30 21:53:48.558155 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:48.558125 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-k6246"] Oct 30 21:53:48.561669 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:48.561648 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-k6246" Oct 30 21:53:48.639919 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:48.639897 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 21:53:48.639919 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:48.639897 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9pxqg\"" Oct 30 21:53:48.667883 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:48.667861 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-k6246"] Oct 30 21:53:48.699774 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:48.699728 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e65cd6f7-2bfd-4b15-ba69-4175ab70a218-cert\") pod \"kserve-controller-manager-776cc66d47-k6246\" (UID: \"e65cd6f7-2bfd-4b15-ba69-4175ab70a218\") " pod="opendatahub/kserve-controller-manager-776cc66d47-k6246" Oct 30 21:53:48.700640 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:48.699937 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jtpm\" (UniqueName: \"kubernetes.io/projected/e65cd6f7-2bfd-4b15-ba69-4175ab70a218-kube-api-access-7jtpm\") pod \"kserve-controller-manager-776cc66d47-k6246\" (UID: \"e65cd6f7-2bfd-4b15-ba69-4175ab70a218\") " pod="opendatahub/kserve-controller-manager-776cc66d47-k6246" Oct 30 21:53:48.800606 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:48.800576 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jtpm\" (UniqueName: \"kubernetes.io/projected/e65cd6f7-2bfd-4b15-ba69-4175ab70a218-kube-api-access-7jtpm\") pod \"kserve-controller-manager-776cc66d47-k6246\" (UID: \"e65cd6f7-2bfd-4b15-ba69-4175ab70a218\") " pod="opendatahub/kserve-controller-manager-776cc66d47-k6246" Oct 30 21:53:48.800761 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:48.800690 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e65cd6f7-2bfd-4b15-ba69-4175ab70a218-cert\") pod \"kserve-controller-manager-776cc66d47-k6246\" (UID: \"e65cd6f7-2bfd-4b15-ba69-4175ab70a218\") " pod="opendatahub/kserve-controller-manager-776cc66d47-k6246" Oct 30 21:53:48.803208 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:48.803185 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e65cd6f7-2bfd-4b15-ba69-4175ab70a218-cert\") pod \"kserve-controller-manager-776cc66d47-k6246\" (UID: \"e65cd6f7-2bfd-4b15-ba69-4175ab70a218\") " pod="opendatahub/kserve-controller-manager-776cc66d47-k6246" Oct 30 21:53:48.874219 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:48.874159 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jtpm\" (UniqueName: \"kubernetes.io/projected/e65cd6f7-2bfd-4b15-ba69-4175ab70a218-kube-api-access-7jtpm\") pod \"kserve-controller-manager-776cc66d47-k6246\" (UID: \"e65cd6f7-2bfd-4b15-ba69-4175ab70a218\") " pod="opendatahub/kserve-controller-manager-776cc66d47-k6246" Oct 30 21:53:48.875010 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:48.874987 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-k6246" Oct 30 21:53:48.920375 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:48.920338 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xg5lh"] Oct 30 21:53:48.925449 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:48.925426 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-xg5lh" Oct 30 21:53:48.983371 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:48.983329 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 21:53:48.983524 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:48.983395 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-whtf5\"" Oct 30 21:53:48.992085 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:48.992059 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xg5lh"] Oct 30 21:53:49.001960 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:49.001938 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t29c\" (UniqueName: \"kubernetes.io/projected/2453d2d7-4fa2-4f97-91f8-ab58457daeb2-kube-api-access-7t29c\") pod \"model-serving-api-75bd89fdfc-xg5lh\" (UID: \"2453d2d7-4fa2-4f97-91f8-ab58457daeb2\") " pod="opendatahub/model-serving-api-75bd89fdfc-xg5lh" Oct 30 21:53:49.002085 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:49.001994 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2453d2d7-4fa2-4f97-91f8-ab58457daeb2-tls-certs\") pod \"model-serving-api-75bd89fdfc-xg5lh\" (UID: \"2453d2d7-4fa2-4f97-91f8-ab58457daeb2\") " pod="opendatahub/model-serving-api-75bd89fdfc-xg5lh" Oct 30 21:53:49.103080 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:49.103054 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t29c\" (UniqueName: \"kubernetes.io/projected/2453d2d7-4fa2-4f97-91f8-ab58457daeb2-kube-api-access-7t29c\") pod \"model-serving-api-75bd89fdfc-xg5lh\" (UID: \"2453d2d7-4fa2-4f97-91f8-ab58457daeb2\") " pod="opendatahub/model-serving-api-75bd89fdfc-xg5lh" Oct 30 21:53:49.103237 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:49.103108 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2453d2d7-4fa2-4f97-91f8-ab58457daeb2-tls-certs\") pod \"model-serving-api-75bd89fdfc-xg5lh\" (UID: \"2453d2d7-4fa2-4f97-91f8-ab58457daeb2\") " pod="opendatahub/model-serving-api-75bd89fdfc-xg5lh" Oct 30 21:53:49.105581 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:49.105556 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2453d2d7-4fa2-4f97-91f8-ab58457daeb2-tls-certs\") pod \"model-serving-api-75bd89fdfc-xg5lh\" (UID: \"2453d2d7-4fa2-4f97-91f8-ab58457daeb2\") " pod="opendatahub/model-serving-api-75bd89fdfc-xg5lh" Oct 30 21:53:49.164123 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:49.164069 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t29c\" (UniqueName: \"kubernetes.io/projected/2453d2d7-4fa2-4f97-91f8-ab58457daeb2-kube-api-access-7t29c\") pod \"model-serving-api-75bd89fdfc-xg5lh\" (UID: \"2453d2d7-4fa2-4f97-91f8-ab58457daeb2\") " pod="opendatahub/model-serving-api-75bd89fdfc-xg5lh" Oct 30 21:53:49.244121 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:49.244097 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-xg5lh" Oct 30 21:53:49.262825 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:49.262802 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-k6246"] Oct 30 21:53:49.264818 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:53:49.264796 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65cd6f7_2bfd_4b15_ba69_4175ab70a218.slice/crio-0ff134496372893248277d603ecf7d47050cef3338da98f5c5d26ed835fba391 WatchSource:0}: Error finding container 0ff134496372893248277d603ecf7d47050cef3338da98f5c5d26ed835fba391: Status 404 returned error can't find the container with id 0ff134496372893248277d603ecf7d47050cef3338da98f5c5d26ed835fba391 Oct 30 21:53:49.620757 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:49.620732 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xg5lh"] Oct 30 21:53:49.622216 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:53:49.622190 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2453d2d7_4fa2_4f97_91f8_ab58457daeb2.slice/crio-c6f02774c06bd96a9f7fe11ac698c106033c1d28102d7ae0d73de27a9d963195 WatchSource:0}: Error finding container c6f02774c06bd96a9f7fe11ac698c106033c1d28102d7ae0d73de27a9d963195: Status 404 returned error can't find the container with id c6f02774c06bd96a9f7fe11ac698c106033c1d28102d7ae0d73de27a9d963195 Oct 30 21:53:50.108934 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:50.108904 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-xg5lh" event={"ID":"2453d2d7-4fa2-4f97-91f8-ab58457daeb2","Type":"ContainerStarted","Data":"c6f02774c06bd96a9f7fe11ac698c106033c1d28102d7ae0d73de27a9d963195"} Oct 30 21:53:50.110581 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:50.110561 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-k6246" event={"ID":"e65cd6f7-2bfd-4b15-ba69-4175ab70a218","Type":"ContainerStarted","Data":"0ff134496372893248277d603ecf7d47050cef3338da98f5c5d26ed835fba391"} Oct 30 21:53:52.124361 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:52.124320 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-xg5lh" event={"ID":"2453d2d7-4fa2-4f97-91f8-ab58457daeb2","Type":"ContainerStarted","Data":"8949bc20b90e0261155350202f73f4318f379837b6b806297af237b6d117c157"} Oct 30 21:53:52.124761 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:52.124516 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-xg5lh" Oct 30 21:53:52.255676 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:52.255627 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-xg5lh" podStartSLOduration=1.85496952 podStartE2EDuration="4.255597931s" totalImagesPullingTime="2.400628411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:52.251279895 +0000 UTC m=+565.351711627" watchObservedRunningTime="2026-10-30 21:53:52.255597931 +0000 UTC m=+565.356029664" Oct 30 21:53:53.074029 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:53.074003 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-7m8xr" Oct 30 21:53:53.130736 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:53.130701 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-k6246" event={"ID":"e65cd6f7-2bfd-4b15-ba69-4175ab70a218","Type":"ContainerStarted","Data":"3d17c32a57f1a4ab353a250aa5ddf909e7cfcb68e9e2b68248d436a5c3afbedc"} Oct 30 21:53:53.131074 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:53.130776 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-k6246" Oct 30 21:53:53.321769 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:53.321643 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-k6246" podStartSLOduration=1.938360069 podStartE2EDuration="5.32160265s" totalImagesPullingTime="3.383242581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:53.321551012 +0000 UTC m=+566.421982746" watchObservedRunningTime="2026-10-30 21:53:53.32160265 +0000 UTC m=+566.422034382" Oct 30 21:53:54.080264 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:54.080236 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bb56545f7-h6p47" Oct 30 21:53:54.252969 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:54.252939 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-7m8xr"] Oct 30 21:53:54.253424 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:54.253154 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-7m8xr" podUID="38a56e90-ae42-4011-9ae4-f13df5fd5254" containerName="manager" containerID="cri-o://beb94ab5922b64f4d97ec21f236bcd996dbc7b1400d8be302f385e65dee3a6f5" gracePeriod=30 Oct 30 21:53:54.490784 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:54.490759 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-7m8xr" Oct 30 21:53:54.650514 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:54.650459 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38a56e90-ae42-4011-9ae4-f13df5fd5254-webhook-cert\" (UniqueName: \"kubernetes.io/secret/38a56e90-ae42-4011-9ae4-f13df5fd5254-webhook-cert\") pod \"38a56e90-ae42-4011-9ae4-f13df5fd5254\" (UID: \"38a56e90-ae42-4011-9ae4-f13df5fd5254\") " Oct 30 21:53:54.650514 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:54.650495 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a56e90-ae42-4011-9ae4-f13df5fd5254-kube-api-access-k7k7j\" (UniqueName: \"kubernetes.io/projected/38a56e90-ae42-4011-9ae4-f13df5fd5254-kube-api-access-k7k7j\") pod \"38a56e90-ae42-4011-9ae4-f13df5fd5254\" (UID: \"38a56e90-ae42-4011-9ae4-f13df5fd5254\") " Oct 30 21:53:54.650691 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:54.650575 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38a56e90-ae42-4011-9ae4-f13df5fd5254-metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a56e90-ae42-4011-9ae4-f13df5fd5254-metrics-certs\") pod \"38a56e90-ae42-4011-9ae4-f13df5fd5254\" (UID: \"38a56e90-ae42-4011-9ae4-f13df5fd5254\") " Oct 30 21:53:54.652747 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:54.652718 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38a56e90-ae42-4011-9ae4-f13df5fd5254-webhook-cert" pod "38a56e90-ae42-4011-9ae4-f13df5fd5254" (UID: "38a56e90-ae42-4011-9ae4-f13df5fd5254"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:54.652847 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:54.652752 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a56e90-ae42-4011-9ae4-f13df5fd5254-kube-api-access-k7k7j" pod "38a56e90-ae42-4011-9ae4-f13df5fd5254" (UID: "38a56e90-ae42-4011-9ae4-f13df5fd5254"). InnerVolumeSpecName "kube-api-access-k7k7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:54.652894 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:54.652845 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38a56e90-ae42-4011-9ae4-f13df5fd5254-metrics-certs" pod "38a56e90-ae42-4011-9ae4-f13df5fd5254" (UID: "38a56e90-ae42-4011-9ae4-f13df5fd5254"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:54.751635 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:54.751587 4335 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a56e90-ae42-4011-9ae4-f13df5fd5254-metrics-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:54.751635 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:54.751635 4335 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38a56e90-ae42-4011-9ae4-f13df5fd5254-webhook-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:54.751769 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:54.751648 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7k7j\" (UniqueName: \"kubernetes.io/projected/38a56e90-ae42-4011-9ae4-f13df5fd5254-kube-api-access-k7k7j\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:55.142540 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:55.142513 4335 generic.go:356] "Generic (PLEG): container finished" podID="38a56e90-ae42-4011-9ae4-f13df5fd5254" containerID="beb94ab5922b64f4d97ec21f236bcd996dbc7b1400d8be302f385e65dee3a6f5" exitCode=0 Oct 30 21:53:55.142670 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:55.142564 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-7m8xr" Oct 30 21:53:55.142670 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:55.142599 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-7m8xr" event={"ID":"38a56e90-ae42-4011-9ae4-f13df5fd5254","Type":"ContainerDied","Data":"beb94ab5922b64f4d97ec21f236bcd996dbc7b1400d8be302f385e65dee3a6f5"} Oct 30 21:53:55.142670 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:55.142663 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-7m8xr" event={"ID":"38a56e90-ae42-4011-9ae4-f13df5fd5254","Type":"ContainerDied","Data":"025a2c5c291d6b98e98eb70de525ae6cebf704de14548200911251256fec622f"} Oct 30 21:53:55.142780 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:55.142689 4335 scope.go:122] "RemoveContainer" containerID="beb94ab5922b64f4d97ec21f236bcd996dbc7b1400d8be302f385e65dee3a6f5" Oct 30 21:53:55.151476 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:55.151458 4335 scope.go:122] "RemoveContainer" containerID="beb94ab5922b64f4d97ec21f236bcd996dbc7b1400d8be302f385e65dee3a6f5" Oct 30 21:53:55.151772 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:53:55.151754 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb94ab5922b64f4d97ec21f236bcd996dbc7b1400d8be302f385e65dee3a6f5\": container with ID starting with beb94ab5922b64f4d97ec21f236bcd996dbc7b1400d8be302f385e65dee3a6f5 not found: ID does not exist" containerID="beb94ab5922b64f4d97ec21f236bcd996dbc7b1400d8be302f385e65dee3a6f5" Oct 30 21:53:55.151823 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:55.151780 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb94ab5922b64f4d97ec21f236bcd996dbc7b1400d8be302f385e65dee3a6f5"} err="failed to get container status \"beb94ab5922b64f4d97ec21f236bcd996dbc7b1400d8be302f385e65dee3a6f5\": rpc error: code = NotFound desc = could not find container \"beb94ab5922b64f4d97ec21f236bcd996dbc7b1400d8be302f385e65dee3a6f5\": container with ID starting with beb94ab5922b64f4d97ec21f236bcd996dbc7b1400d8be302f385e65dee3a6f5 not found: ID does not exist" Oct 30 21:53:55.308468 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:55.308440 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-7m8xr"] Oct 30 21:53:55.318929 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:53:55.318868 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:53:55.359028 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:55.358962 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-7m8xr"] Oct 30 21:53:57.061486 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:53:57.061449 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a56e90-ae42-4011-9ae4-f13df5fd5254" path="/var/lib/kubelet/pods/38a56e90-ae42-4011-9ae4-f13df5fd5254/volumes" Oct 30 21:54:03.134855 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:03.134821 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-xg5lh" Oct 30 21:54:23.448209 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:23.448178 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 21:54:24.141424 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:24.141392 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-k6246" Oct 30 21:54:25.342303 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:54:25.342229 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:54:27.076777 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:27.076747 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 21:54:27.078497 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:27.078477 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 21:54:32.366024 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:32.365988 4335 generic.go:356] "Generic (PLEG): container finished" podID="91fba568-9213-4aa9-a093-17df3f0c506e" containerID="f091890b79771482e31d2f6a353e6e698c91a23f792aae79e8ffcce605ab2441" exitCode=0 Oct 30 21:54:32.366385 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:32.366064 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" event={"ID":"91fba568-9213-4aa9-a093-17df3f0c506e","Type":"ContainerDied","Data":"f091890b79771482e31d2f6a353e6e698c91a23f792aae79e8ffcce605ab2441"} Oct 30 21:54:32.366429 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:32.366417 4335 scope.go:122] "RemoveContainer" containerID="f091890b79771482e31d2f6a353e6e698c91a23f792aae79e8ffcce605ab2441" Oct 30 21:54:32.367572 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:32.367545 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:32.367666 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:32.367646 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:32.367707 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:32.367692 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:33.373480 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:33.373442 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" event={"ID":"91fba568-9213-4aa9-a093-17df3f0c506e","Type":"ContainerStarted","Data":"3eb1c251b5a4ccbd3adf3a07e097d1a7388b40950d17b3cc5eafdd045a984d88"} Oct 30 21:54:33.577896 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:33.577860 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:54:33.578067 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:33.578049 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hbnxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 21:54:33.578141 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:33.578112 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" podUID="91fba568-9213-4aa9-a093-17df3f0c506e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 21:54:34.578510 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:34.578482 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hbnxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 21:54:34.578895 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:34.578544 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" podUID="91fba568-9213-4aa9-a093-17df3f0c506e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 21:54:35.578295 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:35.578256 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hbnxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 21:54:35.578461 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:35.578320 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" podUID="91fba568-9213-4aa9-a093-17df3f0c506e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 21:54:36.577759 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:36.577725 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hbnxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 21:54:36.578136 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:36.577786 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" podUID="91fba568-9213-4aa9-a093-17df3f0c506e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 21:54:37.578714 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:37.578683 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hbnxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 21:54:37.579089 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:37.578741 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" podUID="91fba568-9213-4aa9-a093-17df3f0c506e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 21:54:38.582178 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:38.582149 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:54:38.582653 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:38.582606 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:54:38.583291 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:54:38.583275 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 21:54:55.320085 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:54:55.320042 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:55:25.318388 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:55:25.318353 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:55:55.320481 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:55:55.320402 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:55:55.523075 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.523035 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm"] Oct 30 21:55:55.523465 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.523445 4335 cpu_manager.go:414] "containerMap: removing container" podUID="38a56e90-ae42-4011-9ae4-f13df5fd5254" containerName="manager" Oct 30 21:55:55.523553 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.523467 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a56e90-ae42-4011-9ae4-f13df5fd5254" containerName="manager" Oct 30 21:55:55.523608 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.523565 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a56e90-ae42-4011-9ae4-f13df5fd5254" containerName="manager" Oct 30 21:55:55.526439 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.526416 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.611295 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.611240 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c054348-505a-4763-876b-5447d401e0ec-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm\" (UID: \"2c054348-505a-4763-876b-5447d401e0ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.611404 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.611298 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c054348-505a-4763-876b-5447d401e0ec-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm\" (UID: \"2c054348-505a-4763-876b-5447d401e0ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.611404 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.611359 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c054348-505a-4763-876b-5447d401e0ec-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm\" (UID: \"2c054348-505a-4763-876b-5447d401e0ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.611404 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.611377 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg2l4\" (UniqueName: \"kubernetes.io/projected/2c054348-505a-4763-876b-5447d401e0ec-kube-api-access-kg2l4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm\" (UID: \"2c054348-505a-4763-876b-5447d401e0ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.611597 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.611408 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c054348-505a-4763-876b-5447d401e0ec-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm\" (UID: \"2c054348-505a-4763-876b-5447d401e0ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.611597 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.611425 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c054348-505a-4763-876b-5447d401e0ec-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm\" (UID: \"2c054348-505a-4763-876b-5447d401e0ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.627957 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.627934 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 21:55:55.628185 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.628164 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m2997\"" Oct 30 21:55:55.628284 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.628191 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 21:55:55.628284 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.628240 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 21:55:55.630772 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.630752 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm"] Oct 30 21:55:55.712548 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.712522 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c054348-505a-4763-876b-5447d401e0ec-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm\" (UID: \"2c054348-505a-4763-876b-5447d401e0ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.712695 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.712554 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c054348-505a-4763-876b-5447d401e0ec-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm\" (UID: \"2c054348-505a-4763-876b-5447d401e0ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.712695 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.712572 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg2l4\" (UniqueName: \"kubernetes.io/projected/2c054348-505a-4763-876b-5447d401e0ec-kube-api-access-kg2l4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm\" (UID: \"2c054348-505a-4763-876b-5447d401e0ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.712831 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.712807 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c054348-505a-4763-876b-5447d401e0ec-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm\" (UID: \"2c054348-505a-4763-876b-5447d401e0ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.712876 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.712859 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c054348-505a-4763-876b-5447d401e0ec-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm\" (UID: \"2c054348-505a-4763-876b-5447d401e0ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.712919 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.712903 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c054348-505a-4763-876b-5447d401e0ec-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm\" (UID: \"2c054348-505a-4763-876b-5447d401e0ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.713093 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.713068 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c054348-505a-4763-876b-5447d401e0ec-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm\" (UID: \"2c054348-505a-4763-876b-5447d401e0ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.713219 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.713133 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c054348-505a-4763-876b-5447d401e0ec-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm\" (UID: \"2c054348-505a-4763-876b-5447d401e0ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.713308 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.713287 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c054348-505a-4763-876b-5447d401e0ec-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm\" (UID: \"2c054348-505a-4763-876b-5447d401e0ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.714981 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.714959 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c054348-505a-4763-876b-5447d401e0ec-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm\" (UID: \"2c054348-505a-4763-876b-5447d401e0ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.715144 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.715127 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c054348-505a-4763-876b-5447d401e0ec-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm\" (UID: \"2c054348-505a-4763-876b-5447d401e0ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.779591 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.779554 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg2l4\" (UniqueName: \"kubernetes.io/projected/2c054348-505a-4763-876b-5447d401e0ec-kube-api-access-kg2l4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm\" (UID: \"2c054348-505a-4763-876b-5447d401e0ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:55.840108 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:55.840083 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:56.215258 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:56.215236 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm"] Oct 30 21:55:56.217447 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:55:56.217425 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c054348_505a_4763_876b_5447d401e0ec.slice/crio-8a9cc7fc75c3568f584ed81e530f710b1d8fe8e65265087bd069551c8279d292 WatchSource:0}: Error finding container 8a9cc7fc75c3568f584ed81e530f710b1d8fe8e65265087bd069551c8279d292: Status 404 returned error can't find the container with id 8a9cc7fc75c3568f584ed81e530f710b1d8fe8e65265087bd069551c8279d292 Oct 30 21:55:56.218797 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:56.218782 4335 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:55:56.830962 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:56.830929 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" event={"ID":"2c054348-505a-4763-876b-5447d401e0ec","Type":"ContainerStarted","Data":"8a9cc7fc75c3568f584ed81e530f710b1d8fe8e65265087bd069551c8279d292"} Oct 30 21:55:57.839440 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:57.839401 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" event={"ID":"2c054348-505a-4763-876b-5447d401e0ec","Type":"ContainerStarted","Data":"2e26e514eb8c3b9497117a774672d14dbce70158bbe5da1da29b6aacfdbea72d"} Oct 30 21:55:57.840696 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:57.840670 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:55:57.957073 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:57.957032 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" podStartSLOduration=1.439519652 podStartE2EDuration="2.957018439s" totalImagesPullingTime="1.517498787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:57.955907444 +0000 UTC m=+691.056339176" watchObservedRunningTime="2026-10-30 21:55:57.957018439 +0000 UTC m=+691.057450170" Oct 30 21:55:59.143528 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:55:59.143492 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 21:56:05.841125 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:05.841090 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:56:05.853587 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:05.853561 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:56:05.890640 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:05.890604 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm" Oct 30 21:56:07.527573 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:07.527542 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 21:56:08.052376 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.052343 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g"] Oct 30 21:56:08.058259 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.058235 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.120329 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.120300 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 21:56:08.165156 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.165116 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g"] Oct 30 21:56:08.214279 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.214237 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldx56\" (UniqueName: \"kubernetes.io/projected/e472aa82-daee-470a-a544-bd40adc2073b-kube-api-access-ldx56\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j7l8g\" (UID: \"e472aa82-daee-470a-a544-bd40adc2073b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.214279 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.214273 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e472aa82-daee-470a-a544-bd40adc2073b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j7l8g\" (UID: \"e472aa82-daee-470a-a544-bd40adc2073b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.214513 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.214333 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e472aa82-daee-470a-a544-bd40adc2073b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j7l8g\" (UID: \"e472aa82-daee-470a-a544-bd40adc2073b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.214513 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.214415 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e472aa82-daee-470a-a544-bd40adc2073b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j7l8g\" (UID: \"e472aa82-daee-470a-a544-bd40adc2073b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.214673 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.214599 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e472aa82-daee-470a-a544-bd40adc2073b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j7l8g\" (UID: \"e472aa82-daee-470a-a544-bd40adc2073b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.214741 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.214666 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e472aa82-daee-470a-a544-bd40adc2073b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j7l8g\" (UID: \"e472aa82-daee-470a-a544-bd40adc2073b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.315532 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.315454 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldx56\" (UniqueName: \"kubernetes.io/projected/e472aa82-daee-470a-a544-bd40adc2073b-kube-api-access-ldx56\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j7l8g\" (UID: \"e472aa82-daee-470a-a544-bd40adc2073b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.315532 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.315499 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e472aa82-daee-470a-a544-bd40adc2073b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j7l8g\" (UID: \"e472aa82-daee-470a-a544-bd40adc2073b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.315764 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.315538 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e472aa82-daee-470a-a544-bd40adc2073b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j7l8g\" (UID: \"e472aa82-daee-470a-a544-bd40adc2073b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.315764 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.315569 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e472aa82-daee-470a-a544-bd40adc2073b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j7l8g\" (UID: \"e472aa82-daee-470a-a544-bd40adc2073b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.315764 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.315606 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e472aa82-daee-470a-a544-bd40adc2073b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j7l8g\" (UID: \"e472aa82-daee-470a-a544-bd40adc2073b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.315764 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.315651 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e472aa82-daee-470a-a544-bd40adc2073b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j7l8g\" (UID: \"e472aa82-daee-470a-a544-bd40adc2073b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.316018 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.315974 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e472aa82-daee-470a-a544-bd40adc2073b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j7l8g\" (UID: \"e472aa82-daee-470a-a544-bd40adc2073b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.316080 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.316050 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e472aa82-daee-470a-a544-bd40adc2073b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j7l8g\" (UID: \"e472aa82-daee-470a-a544-bd40adc2073b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.316344 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.316321 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e472aa82-daee-470a-a544-bd40adc2073b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j7l8g\" (UID: \"e472aa82-daee-470a-a544-bd40adc2073b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.319292 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.319269 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e472aa82-daee-470a-a544-bd40adc2073b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j7l8g\" (UID: \"e472aa82-daee-470a-a544-bd40adc2073b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.328221 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.328192 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e472aa82-daee-470a-a544-bd40adc2073b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j7l8g\" (UID: \"e472aa82-daee-470a-a544-bd40adc2073b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.376939 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.376902 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldx56\" (UniqueName: \"kubernetes.io/projected/e472aa82-daee-470a-a544-bd40adc2073b-kube-api-access-ldx56\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j7l8g\" (UID: \"e472aa82-daee-470a-a544-bd40adc2073b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:08.672856 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:08.672780 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:09.051294 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:56:09.051268 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode472aa82_daee_470a_a544_bd40adc2073b.slice/crio-3b06e91e02a9e66ea1d0cd6f7b86205dd8cade5655c7d6ef12f8107f3381cdcb WatchSource:0}: Error finding container 3b06e91e02a9e66ea1d0cd6f7b86205dd8cade5655c7d6ef12f8107f3381cdcb: Status 404 returned error can't find the container with id 3b06e91e02a9e66ea1d0cd6f7b86205dd8cade5655c7d6ef12f8107f3381cdcb Oct 30 21:56:09.055656 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:09.055636 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g"] Oct 30 21:56:09.904366 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:09.904335 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" event={"ID":"e472aa82-daee-470a-a544-bd40adc2073b","Type":"ContainerStarted","Data":"54104583b42d5949d1cd041184e8f350a9e145be655bb970c76786bc12d7a990"} Oct 30 21:56:09.904771 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:09.904385 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" event={"ID":"e472aa82-daee-470a-a544-bd40adc2073b","Type":"ContainerStarted","Data":"3b06e91e02a9e66ea1d0cd6f7b86205dd8cade5655c7d6ef12f8107f3381cdcb"} Oct 30 21:56:10.023298 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:10.023215 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" podStartSLOduration=1.413177157 podStartE2EDuration="2.023198828s" totalImagesPullingTime="610.021671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:10.021640251 +0000 UTC m=+703.122071983" watchObservedRunningTime="2026-10-30 21:56:10.023198828 +0000 UTC m=+703.123630561" Oct 30 21:56:10.673627 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:10.673577 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:16.419898 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:16.419859 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 21:56:18.368955 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.368925 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-tpwc7"] Oct 30 21:56:18.372857 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.372840 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.401343 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.401312 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3c54c2-eb66-4b04-b6ef-55c1691873b9-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-tpwc7\" (UID: \"3e3c54c2-eb66-4b04-b6ef-55c1691873b9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.401497 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.401351 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3c54c2-eb66-4b04-b6ef-55c1691873b9-home\") pod \"echo-simulator-kserve-64b6f8f96c-tpwc7\" (UID: \"3e3c54c2-eb66-4b04-b6ef-55c1691873b9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.401497 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.401381 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmxx4\" (UniqueName: \"kubernetes.io/projected/3e3c54c2-eb66-4b04-b6ef-55c1691873b9-kube-api-access-rmxx4\") pod \"echo-simulator-kserve-64b6f8f96c-tpwc7\" (UID: \"3e3c54c2-eb66-4b04-b6ef-55c1691873b9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.401645 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.401540 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3c54c2-eb66-4b04-b6ef-55c1691873b9-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-tpwc7\" (UID: \"3e3c54c2-eb66-4b04-b6ef-55c1691873b9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.401645 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.401567 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3c54c2-eb66-4b04-b6ef-55c1691873b9-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-tpwc7\" (UID: \"3e3c54c2-eb66-4b04-b6ef-55c1691873b9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.401645 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.401584 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3c54c2-eb66-4b04-b6ef-55c1691873b9-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-tpwc7\" (UID: \"3e3c54c2-eb66-4b04-b6ef-55c1691873b9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.473756 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.473733 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 21:56:18.476284 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.476261 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-tpwc7"] Oct 30 21:56:18.502583 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.502561 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmxx4\" (UniqueName: \"kubernetes.io/projected/3e3c54c2-eb66-4b04-b6ef-55c1691873b9-kube-api-access-rmxx4\") pod \"echo-simulator-kserve-64b6f8f96c-tpwc7\" (UID: \"3e3c54c2-eb66-4b04-b6ef-55c1691873b9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.502681 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.502634 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3c54c2-eb66-4b04-b6ef-55c1691873b9-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-tpwc7\" (UID: \"3e3c54c2-eb66-4b04-b6ef-55c1691873b9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.502681 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.502667 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3c54c2-eb66-4b04-b6ef-55c1691873b9-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-tpwc7\" (UID: \"3e3c54c2-eb66-4b04-b6ef-55c1691873b9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.502772 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.502689 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3c54c2-eb66-4b04-b6ef-55c1691873b9-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-tpwc7\" (UID: \"3e3c54c2-eb66-4b04-b6ef-55c1691873b9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.502772 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.502714 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3c54c2-eb66-4b04-b6ef-55c1691873b9-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-tpwc7\" (UID: \"3e3c54c2-eb66-4b04-b6ef-55c1691873b9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.502772 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.502741 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3c54c2-eb66-4b04-b6ef-55c1691873b9-home\") pod \"echo-simulator-kserve-64b6f8f96c-tpwc7\" (UID: \"3e3c54c2-eb66-4b04-b6ef-55c1691873b9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.502978 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.502959 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3c54c2-eb66-4b04-b6ef-55c1691873b9-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-tpwc7\" (UID: \"3e3c54c2-eb66-4b04-b6ef-55c1691873b9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.503066 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.503045 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3c54c2-eb66-4b04-b6ef-55c1691873b9-home\") pod \"echo-simulator-kserve-64b6f8f96c-tpwc7\" (UID: \"3e3c54c2-eb66-4b04-b6ef-55c1691873b9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.503127 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.503104 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3c54c2-eb66-4b04-b6ef-55c1691873b9-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-tpwc7\" (UID: \"3e3c54c2-eb66-4b04-b6ef-55c1691873b9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.505002 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.504981 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3c54c2-eb66-4b04-b6ef-55c1691873b9-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-tpwc7\" (UID: \"3e3c54c2-eb66-4b04-b6ef-55c1691873b9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.505333 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.505317 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3c54c2-eb66-4b04-b6ef-55c1691873b9-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-tpwc7\" (UID: \"3e3c54c2-eb66-4b04-b6ef-55c1691873b9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.563940 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.563909 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmxx4\" (UniqueName: \"kubernetes.io/projected/3e3c54c2-eb66-4b04-b6ef-55c1691873b9-kube-api-access-rmxx4\") pod \"echo-simulator-kserve-64b6f8f96c-tpwc7\" (UID: \"3e3c54c2-eb66-4b04-b6ef-55c1691873b9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.673140 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.673066 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:18.686168 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.686135 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:18.686411 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.686372 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:18.966077 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:18.966007 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j7l8g" Oct 30 21:56:19.068859 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:19.068834 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-tpwc7"] Oct 30 21:56:19.070416 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:56:19.070395 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3c54c2_eb66_4b04_b6ef_55c1691873b9.slice/crio-10806c91f052ef60d5f6bd86639645a1634e8636c17158a959e6a5d1b3accb84 WatchSource:0}: Error finding container 10806c91f052ef60d5f6bd86639645a1634e8636c17158a959e6a5d1b3accb84: Status 404 returned error can't find the container with id 10806c91f052ef60d5f6bd86639645a1634e8636c17158a959e6a5d1b3accb84 Oct 30 21:56:19.962643 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:19.961124 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" event={"ID":"3e3c54c2-eb66-4b04-b6ef-55c1691873b9","Type":"ContainerStarted","Data":"10806c91f052ef60d5f6bd86639645a1634e8636c17158a959e6a5d1b3accb84"} Oct 30 21:56:21.975695 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:21.975590 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" event={"ID":"3e3c54c2-eb66-4b04-b6ef-55c1691873b9","Type":"ContainerStarted","Data":"d9e8271934009791e4f80d63885b922bf0b90648d1e74f53e84c9003dfaac46d"} Oct 30 21:56:22.094824 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:22.094775 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" podStartSLOduration=1.501141771 podStartE2EDuration="4.094758848s" totalImagesPullingTime="2.593617077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:22.091938519 +0000 UTC m=+715.192370264" watchObservedRunningTime="2026-10-30 21:56:22.094758848 +0000 UTC m=+715.195190580" Oct 30 21:56:22.686542 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:22.686511 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:25.320134 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:56:25.320096 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:56:25.469258 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.469220 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x"] Oct 30 21:56:25.472890 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.472868 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.543120 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.543099 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 21:56:25.569571 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.569543 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1023f325-f4c6-41ae-86f5-d0796101a380-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x\" (UID: \"1023f325-f4c6-41ae-86f5-d0796101a380\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.569696 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.569575 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1023f325-f4c6-41ae-86f5-d0796101a380-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x\" (UID: \"1023f325-f4c6-41ae-86f5-d0796101a380\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.569696 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.569599 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1023f325-f4c6-41ae-86f5-d0796101a380-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x\" (UID: \"1023f325-f4c6-41ae-86f5-d0796101a380\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.569794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.569771 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1023f325-f4c6-41ae-86f5-d0796101a380-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x\" (UID: \"1023f325-f4c6-41ae-86f5-d0796101a380\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.569829 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.569808 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjrb6\" (UniqueName: \"kubernetes.io/projected/1023f325-f4c6-41ae-86f5-d0796101a380-kube-api-access-vjrb6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x\" (UID: \"1023f325-f4c6-41ae-86f5-d0796101a380\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.569886 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.569869 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1023f325-f4c6-41ae-86f5-d0796101a380-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x\" (UID: \"1023f325-f4c6-41ae-86f5-d0796101a380\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.576521 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.576480 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x"] Oct 30 21:56:25.670682 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.670650 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1023f325-f4c6-41ae-86f5-d0796101a380-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x\" (UID: \"1023f325-f4c6-41ae-86f5-d0796101a380\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.670808 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.670701 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjrb6\" (UniqueName: \"kubernetes.io/projected/1023f325-f4c6-41ae-86f5-d0796101a380-kube-api-access-vjrb6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x\" (UID: \"1023f325-f4c6-41ae-86f5-d0796101a380\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.670808 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.670751 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1023f325-f4c6-41ae-86f5-d0796101a380-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x\" (UID: \"1023f325-f4c6-41ae-86f5-d0796101a380\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.670808 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.670801 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1023f325-f4c6-41ae-86f5-d0796101a380-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x\" (UID: \"1023f325-f4c6-41ae-86f5-d0796101a380\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.670972 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.670834 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1023f325-f4c6-41ae-86f5-d0796101a380-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x\" (UID: \"1023f325-f4c6-41ae-86f5-d0796101a380\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.670972 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.670881 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1023f325-f4c6-41ae-86f5-d0796101a380-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x\" (UID: \"1023f325-f4c6-41ae-86f5-d0796101a380\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.671203 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.671176 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1023f325-f4c6-41ae-86f5-d0796101a380-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x\" (UID: \"1023f325-f4c6-41ae-86f5-d0796101a380\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.671337 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.671317 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1023f325-f4c6-41ae-86f5-d0796101a380-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x\" (UID: \"1023f325-f4c6-41ae-86f5-d0796101a380\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.671337 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.671328 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1023f325-f4c6-41ae-86f5-d0796101a380-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x\" (UID: \"1023f325-f4c6-41ae-86f5-d0796101a380\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.673269 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.673243 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1023f325-f4c6-41ae-86f5-d0796101a380-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x\" (UID: \"1023f325-f4c6-41ae-86f5-d0796101a380\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.673452 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.673434 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1023f325-f4c6-41ae-86f5-d0796101a380-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x\" (UID: \"1023f325-f4c6-41ae-86f5-d0796101a380\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.737967 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.737940 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjrb6\" (UniqueName: \"kubernetes.io/projected/1023f325-f4c6-41ae-86f5-d0796101a380-kube-api-access-vjrb6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x\" (UID: \"1023f325-f4c6-41ae-86f5-d0796101a380\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:25.784093 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:25.784076 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:26.160587 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:26.160566 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x"] Oct 30 21:56:26.162012 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:56:26.161988 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1023f325_f4c6_41ae_86f5_d0796101a380.slice/crio-5cd3a468d51ab5d77bb9fcbfa73aa626ab69a34a14bbb37e1ab74c83243c7abb WatchSource:0}: Error finding container 5cd3a468d51ab5d77bb9fcbfa73aa626ab69a34a14bbb37e1ab74c83243c7abb: Status 404 returned error can't find the container with id 5cd3a468d51ab5d77bb9fcbfa73aa626ab69a34a14bbb37e1ab74c83243c7abb Oct 30 21:56:27.005109 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:27.005030 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" event={"ID":"1023f325-f4c6-41ae-86f5-d0796101a380","Type":"ContainerStarted","Data":"5dd214d2559905d0d5d9c76c6ff303611ba7eabbd5721df16f5e04c432f12810"} Oct 30 21:56:27.005109 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:27.005066 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" event={"ID":"1023f325-f4c6-41ae-86f5-d0796101a380","Type":"ContainerStarted","Data":"5cd3a468d51ab5d77bb9fcbfa73aa626ab69a34a14bbb37e1ab74c83243c7abb"} Oct 30 21:56:27.123254 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:27.123211 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podStartSLOduration=1.537084071 podStartE2EDuration="2.123197698s" totalImagesPullingTime="586.113627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:27.121084187 +0000 UTC m=+720.221515930" watchObservedRunningTime="2026-10-30 21:56:27.123197698 +0000 UTC m=+720.223629465" Oct 30 21:56:27.785077 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:27.785050 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:28.687320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:28.687286 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:28.701313 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:28.701284 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:29.025204 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:29.025135 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 21:56:29.027944 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:29.027923 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" Oct 30 21:56:35.784279 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:35.784244 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:35.796384 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:35.796361 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:36.068750 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:36.068724 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 21:56:41.529422 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.529387 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv"] Oct 30 21:56:41.532959 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.532939 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.604130 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.604101 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.604245 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.604145 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mpnz\" (UniqueName: \"kubernetes.io/projected/ba8a2354-02ca-46cf-bea7-d97781a505f5-kube-api-access-8mpnz\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.604245 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.604218 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.604324 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.604249 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.604324 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.604267 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.604324 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.604283 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8a2354-02ca-46cf-bea7-d97781a505f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.604421 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.604405 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.604458 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.604431 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.604491 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.604478 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.631667 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.631645 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv"] Oct 30 21:56:41.705533 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.705509 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.705688 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.705550 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.705688 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.705570 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mpnz\" (UniqueName: \"kubernetes.io/projected/ba8a2354-02ca-46cf-bea7-d97781a505f5-kube-api-access-8mpnz\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.705688 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.705589 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.705688 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.705646 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.705688 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.705673 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.705688 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.705690 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8a2354-02ca-46cf-bea7-d97781a505f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.706036 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.705734 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.706036 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.705757 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.706036 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.705970 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.706193 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.706073 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.706193 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.706153 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.706320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.706194 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.706449 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.706426 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8a2354-02ca-46cf-bea7-d97781a505f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.707962 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.707946 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.708265 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.708246 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.766389 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.766363 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.766733 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.766713 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mpnz\" (UniqueName: \"kubernetes.io/projected/ba8a2354-02ca-46cf-bea7-d97781a505f5-kube-api-access-8mpnz\") pod \"maas-default-gateway-openshift-default-6cc458d887-sbvnv\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:41.846989 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:41.846967 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:42.276823 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:42.276800 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv"] Oct 30 21:56:42.279932 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:56:42.279905 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8a2354_02ca_46cf_bea7_d97781a505f5.slice/crio-719326c5e859925ef0a76e319e46230123161afff4f7092179c4fb2388b5ffec WatchSource:0}: Error finding container 719326c5e859925ef0a76e319e46230123161afff4f7092179c4fb2388b5ffec: Status 404 returned error can't find the container with id 719326c5e859925ef0a76e319e46230123161afff4f7092179c4fb2388b5ffec Oct 30 21:56:42.282048 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:42.282016 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:56:42.282126 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:42.282081 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:56:42.282126 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:42.282111 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:56:43.103638 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:43.095790 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" event={"ID":"ba8a2354-02ca-46cf-bea7-d97781a505f5","Type":"ContainerStarted","Data":"f30deb5e197025c9592cdcc8cb863bb149cef7a08645591e2ef2a9d16fce6e14"} Oct 30 21:56:43.103638 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:43.095855 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" event={"ID":"ba8a2354-02ca-46cf-bea7-d97781a505f5","Type":"ContainerStarted","Data":"719326c5e859925ef0a76e319e46230123161afff4f7092179c4fb2388b5ffec"} Oct 30 21:56:43.321071 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:43.320997 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" podStartSLOduration=2.32097529 podStartE2EDuration="2.32097529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:43.267792948 +0000 UTC m=+736.368224682" watchObservedRunningTime="2026-10-30 21:56:43.32097529 +0000 UTC m=+736.421407025" Oct 30 21:56:43.847760 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:43.847729 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:43.849183 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:43.849153 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sbvnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:56:43.849316 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:43.849200 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" podUID="ba8a2354-02ca-46cf-bea7-d97781a505f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 21:56:44.847823 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:44.847792 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sbvnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:56:44.848187 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:44.847841 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" podUID="ba8a2354-02ca-46cf-bea7-d97781a505f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 21:56:45.847376 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:45.847340 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sbvnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:56:45.847552 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:45.847413 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" podUID="ba8a2354-02ca-46cf-bea7-d97781a505f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 21:56:46.848058 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:46.848023 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sbvnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:56:46.848453 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:46.848076 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" podUID="ba8a2354-02ca-46cf-bea7-d97781a505f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 21:56:47.847399 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:47.847363 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sbvnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:56:47.847565 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:47.847417 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" podUID="ba8a2354-02ca-46cf-bea7-d97781a505f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 21:56:48.847632 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:48.847582 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sbvnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:56:48.848012 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:48.847653 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" podUID="ba8a2354-02ca-46cf-bea7-d97781a505f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 21:56:50.486952 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:50.486921 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:50.487433 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:50.487216 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:50.488037 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:50.488009 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 21:56:52.737587 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:52.737549 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-z5wtt"] Oct 30 21:56:52.741962 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:52.741943 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-z5wtt" Oct 30 21:56:52.796551 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:52.796520 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 21:56:52.801485 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:52.801460 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g89n\" (UniqueName: \"kubernetes.io/projected/da0597f7-5b3f-4707-b604-28585e478b00-kube-api-access-6g89n\") pod \"ai-gateway-controller-66dd76bf94-z5wtt\" (UID: \"da0597f7-5b3f-4707-b604-28585e478b00\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-z5wtt" Oct 30 21:56:52.801572 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:52.801500 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0597f7-5b3f-4707-b604-28585e478b00-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-z5wtt\" (UID: \"da0597f7-5b3f-4707-b604-28585e478b00\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-z5wtt" Oct 30 21:56:52.845910 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:52.845885 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 21:56:52.846071 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:52.845958 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-7vlp5\"" Oct 30 21:56:52.848974 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:52.848949 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-z5wtt"] Oct 30 21:56:52.902832 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:52.902806 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0597f7-5b3f-4707-b604-28585e478b00-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-z5wtt\" (UID: \"da0597f7-5b3f-4707-b604-28585e478b00\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-z5wtt" Oct 30 21:56:52.902958 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:52.902941 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g89n\" (UniqueName: \"kubernetes.io/projected/da0597f7-5b3f-4707-b604-28585e478b00-kube-api-access-6g89n\") pod \"ai-gateway-controller-66dd76bf94-z5wtt\" (UID: \"da0597f7-5b3f-4707-b604-28585e478b00\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-z5wtt" Oct 30 21:56:52.905360 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:52.905336 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0597f7-5b3f-4707-b604-28585e478b00-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-z5wtt\" (UID: \"da0597f7-5b3f-4707-b604-28585e478b00\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-z5wtt" Oct 30 21:56:52.964135 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:52.964104 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g89n\" (UniqueName: \"kubernetes.io/projected/da0597f7-5b3f-4707-b604-28585e478b00-kube-api-access-6g89n\") pod \"ai-gateway-controller-66dd76bf94-z5wtt\" (UID: \"da0597f7-5b3f-4707-b604-28585e478b00\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-z5wtt" Oct 30 21:56:53.054761 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:53.054738 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-z5wtt" Oct 30 21:56:53.432797 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:53.432764 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-z5wtt"] Oct 30 21:56:53.433487 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:56:53.433459 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0597f7_5b3f_4707_b604_28585e478b00.slice/crio-324e8711e0877c5ec665892aabe98d6d531908d65b4c77707fedbbb69af2cd8e WatchSource:0}: Error finding container 324e8711e0877c5ec665892aabe98d6d531908d65b4c77707fedbbb69af2cd8e: Status 404 returned error can't find the container with id 324e8711e0877c5ec665892aabe98d6d531908d65b4c77707fedbbb69af2cd8e Oct 30 21:56:54.180415 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:54.180382 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-z5wtt" event={"ID":"da0597f7-5b3f-4707-b604-28585e478b00","Type":"ContainerStarted","Data":"324e8711e0877c5ec665892aabe98d6d531908d65b4c77707fedbbb69af2cd8e"} Oct 30 21:56:55.326609 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:56:55.326577 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:56:56.194413 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:56.194382 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-z5wtt" event={"ID":"da0597f7-5b3f-4707-b604-28585e478b00","Type":"ContainerStarted","Data":"5e6cd86e624c6706e332bf59675167f5743c45fa712f0e0ad7eac9ea8b0065d5"} Oct 30 21:56:56.194584 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:56.194460 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-66dd76bf94-z5wtt" Oct 30 21:56:56.312010 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:56:56.311967 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-66dd76bf94-z5wtt" podStartSLOduration=2.154156641 podStartE2EDuration="4.311953944s" totalImagesPullingTime="2.157797303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:56.309749585 +0000 UTC m=+749.410181316" watchObservedRunningTime="2026-10-30 21:56:56.311953944 +0000 UTC m=+749.412385675" Oct 30 21:57:00.437894 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.437854 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qlsfm"] Oct 30 21:57:00.441824 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.441800 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qlsfm" Oct 30 21:57:00.499646 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.499597 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qlsfm"] Oct 30 21:57:00.545673 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.545643 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nrnk5\"" Oct 30 21:57:00.545838 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.545674 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 21:57:00.566840 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.566808 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxrzw\" (UniqueName: \"kubernetes.io/projected/bdd4f287-fcc7-4b03-b711-dcc0edab30e7-kube-api-access-mxrzw\") pod \"payload-pre-processing-7b98bb678c-qlsfm\" (UID: \"bdd4f287-fcc7-4b03-b711-dcc0edab30e7\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qlsfm" Oct 30 21:57:00.566981 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.566849 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdd4f287-fcc7-4b03-b711-dcc0edab30e7-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qlsfm\" (UID: \"bdd4f287-fcc7-4b03-b711-dcc0edab30e7\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qlsfm" Oct 30 21:57:00.597665 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.597639 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-dbz8f"] Oct 30 21:57:00.601477 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.601453 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-dbz8f" Oct 30 21:57:00.639373 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.639345 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-dbz8f"] Oct 30 21:57:00.667741 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.667710 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxrzw\" (UniqueName: \"kubernetes.io/projected/bdd4f287-fcc7-4b03-b711-dcc0edab30e7-kube-api-access-mxrzw\") pod \"payload-pre-processing-7b98bb678c-qlsfm\" (UID: \"bdd4f287-fcc7-4b03-b711-dcc0edab30e7\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qlsfm" Oct 30 21:57:00.667900 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.667756 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdd4f287-fcc7-4b03-b711-dcc0edab30e7-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qlsfm\" (UID: \"bdd4f287-fcc7-4b03-b711-dcc0edab30e7\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qlsfm" Oct 30 21:57:00.667900 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.667818 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls92l\" (UniqueName: \"kubernetes.io/projected/21dcce2e-c36d-4de3-adb0-b514662b20e5-kube-api-access-ls92l\") pod \"payload-processing-55fbbbf9dc-dbz8f\" (UID: \"21dcce2e-c36d-4de3-adb0-b514662b20e5\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-dbz8f" Oct 30 21:57:00.667900 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.667855 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21dcce2e-c36d-4de3-adb0-b514662b20e5-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-dbz8f\" (UID: \"21dcce2e-c36d-4de3-adb0-b514662b20e5\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-dbz8f" Oct 30 21:57:00.668701 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.668669 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdd4f287-fcc7-4b03-b711-dcc0edab30e7-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qlsfm\" (UID: \"bdd4f287-fcc7-4b03-b711-dcc0edab30e7\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qlsfm" Oct 30 21:57:00.728847 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.728774 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxrzw\" (UniqueName: \"kubernetes.io/projected/bdd4f287-fcc7-4b03-b711-dcc0edab30e7-kube-api-access-mxrzw\") pod \"payload-pre-processing-7b98bb678c-qlsfm\" (UID: \"bdd4f287-fcc7-4b03-b711-dcc0edab30e7\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qlsfm" Oct 30 21:57:00.754417 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.754389 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qlsfm" Oct 30 21:57:00.768989 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.768957 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls92l\" (UniqueName: \"kubernetes.io/projected/21dcce2e-c36d-4de3-adb0-b514662b20e5-kube-api-access-ls92l\") pod \"payload-processing-55fbbbf9dc-dbz8f\" (UID: \"21dcce2e-c36d-4de3-adb0-b514662b20e5\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-dbz8f" Oct 30 21:57:00.769125 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.769011 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21dcce2e-c36d-4de3-adb0-b514662b20e5-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-dbz8f\" (UID: \"21dcce2e-c36d-4de3-adb0-b514662b20e5\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-dbz8f" Oct 30 21:57:00.769902 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.769877 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21dcce2e-c36d-4de3-adb0-b514662b20e5-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-dbz8f\" (UID: \"21dcce2e-c36d-4de3-adb0-b514662b20e5\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-dbz8f" Oct 30 21:57:00.830332 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.830274 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls92l\" (UniqueName: \"kubernetes.io/projected/21dcce2e-c36d-4de3-adb0-b514662b20e5-kube-api-access-ls92l\") pod \"payload-processing-55fbbbf9dc-dbz8f\" (UID: \"21dcce2e-c36d-4de3-adb0-b514662b20e5\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-dbz8f" Oct 30 21:57:00.915702 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:00.915665 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-dbz8f" Oct 30 21:57:01.135571 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:01.135547 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qlsfm"] Oct 30 21:57:01.138372 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:57:01.138332 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd4f287_fcc7_4b03_b711_dcc0edab30e7.slice/crio-99637c12551971080adebd5cfc0019458a7311679616f6171d797111c3456e97 WatchSource:0}: Error finding container 99637c12551971080adebd5cfc0019458a7311679616f6171d797111c3456e97: Status 404 returned error can't find the container with id 99637c12551971080adebd5cfc0019458a7311679616f6171d797111c3456e97 Oct 30 21:57:01.223852 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:01.223822 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qlsfm" event={"ID":"bdd4f287-fcc7-4b03-b711-dcc0edab30e7","Type":"ContainerStarted","Data":"99637c12551971080adebd5cfc0019458a7311679616f6171d797111c3456e97"} Oct 30 21:57:01.304428 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:01.304379 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-dbz8f"] Oct 30 21:57:01.307409 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:57:01.307365 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21dcce2e_c36d_4de3_adb0_b514662b20e5.slice/crio-b0fb544fdef06b607b6dd2568f566c8bc36f519d2d561a46c4fdd91baf014c3b WatchSource:0}: Error finding container b0fb544fdef06b607b6dd2568f566c8bc36f519d2d561a46c4fdd91baf014c3b: Status 404 returned error can't find the container with id b0fb544fdef06b607b6dd2568f566c8bc36f519d2d561a46c4fdd91baf014c3b Oct 30 21:57:02.232776 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:02.232740 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-dbz8f" event={"ID":"21dcce2e-c36d-4de3-adb0-b514662b20e5","Type":"ContainerStarted","Data":"b0fb544fdef06b607b6dd2568f566c8bc36f519d2d561a46c4fdd91baf014c3b"} Oct 30 21:57:04.248209 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:04.248159 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-dbz8f" event={"ID":"21dcce2e-c36d-4de3-adb0-b514662b20e5","Type":"ContainerStarted","Data":"b3795fcdcaab60bd3f593e9a292fea1d6a43777599c807a09e3d45dc38d9814f"} Oct 30 21:57:04.248650 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:04.248284 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-dbz8f" Oct 30 21:57:04.365952 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:04.365911 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-dbz8f" podStartSLOduration=1.619172527 podStartE2EDuration="4.365897121s" totalImagesPullingTime="2.746724594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:04.363228354 +0000 UTC m=+757.463660085" watchObservedRunningTime="2026-10-30 21:57:04.365897121 +0000 UTC m=+757.466329168" Oct 30 21:57:05.260793 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:05.260744 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qlsfm" event={"ID":"bdd4f287-fcc7-4b03-b711-dcc0edab30e7","Type":"ContainerStarted","Data":"507613a6cfbe012b7b3b9156a1a4d46abc2cd3514997dec25a1ce118f0c9f555"} Oct 30 21:57:05.261270 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:05.260848 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qlsfm" Oct 30 21:57:05.429997 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:05.429934 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qlsfm" podStartSLOduration=2.27082008 podStartE2EDuration="5.429921104s" totalImagesPullingTime="3.159101024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:05.427684939 +0000 UTC m=+758.528116670" watchObservedRunningTime="2026-10-30 21:57:05.429921104 +0000 UTC m=+758.530352836" Oct 30 21:57:07.201146 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:07.201110 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-66dd76bf94-z5wtt" Oct 30 21:57:07.565347 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:07.565318 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-fc7c456c9-9z74t"] Oct 30 21:57:07.568962 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:07.568935 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-fc7c456c9-9z74t" Oct 30 21:57:07.626597 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:07.626573 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-fc7c456c9-9z74t"] Oct 30 21:57:07.728510 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:07.728480 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e21a1a66-81c3-4e3f-a37b-5d2c6d799d2e-metrics-certs\") pod \"ai-gateway-controller-fc7c456c9-9z74t\" (UID: \"e21a1a66-81c3-4e3f-a37b-5d2c6d799d2e\") " pod="opendatahub/ai-gateway-controller-fc7c456c9-9z74t" Oct 30 21:57:07.728609 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:07.728541 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c467\" (UniqueName: \"kubernetes.io/projected/e21a1a66-81c3-4e3f-a37b-5d2c6d799d2e-kube-api-access-5c467\") pod \"ai-gateway-controller-fc7c456c9-9z74t\" (UID: \"e21a1a66-81c3-4e3f-a37b-5d2c6d799d2e\") " pod="opendatahub/ai-gateway-controller-fc7c456c9-9z74t" Oct 30 21:57:07.829730 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:07.829674 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c467\" (UniqueName: \"kubernetes.io/projected/e21a1a66-81c3-4e3f-a37b-5d2c6d799d2e-kube-api-access-5c467\") pod \"ai-gateway-controller-fc7c456c9-9z74t\" (UID: \"e21a1a66-81c3-4e3f-a37b-5d2c6d799d2e\") " pod="opendatahub/ai-gateway-controller-fc7c456c9-9z74t" Oct 30 21:57:07.829834 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:07.829734 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e21a1a66-81c3-4e3f-a37b-5d2c6d799d2e-metrics-certs\") pod \"ai-gateway-controller-fc7c456c9-9z74t\" (UID: \"e21a1a66-81c3-4e3f-a37b-5d2c6d799d2e\") " pod="opendatahub/ai-gateway-controller-fc7c456c9-9z74t" Oct 30 21:57:07.832272 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:07.832252 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e21a1a66-81c3-4e3f-a37b-5d2c6d799d2e-metrics-certs\") pod \"ai-gateway-controller-fc7c456c9-9z74t\" (UID: \"e21a1a66-81c3-4e3f-a37b-5d2c6d799d2e\") " pod="opendatahub/ai-gateway-controller-fc7c456c9-9z74t" Oct 30 21:57:07.890300 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:07.890277 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c467\" (UniqueName: \"kubernetes.io/projected/e21a1a66-81c3-4e3f-a37b-5d2c6d799d2e-kube-api-access-5c467\") pod \"ai-gateway-controller-fc7c456c9-9z74t\" (UID: \"e21a1a66-81c3-4e3f-a37b-5d2c6d799d2e\") " pod="opendatahub/ai-gateway-controller-fc7c456c9-9z74t" Oct 30 21:57:08.181029 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:08.180941 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-fc7c456c9-9z74t" Oct 30 21:57:08.554462 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:08.554436 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-fc7c456c9-9z74t"] Oct 30 21:57:08.555798 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:57:08.555773 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21a1a66_81c3_4e3f_a37b_5d2c6d799d2e.slice/crio-7b1028c3af87d55375d3daa25d4f253bcbbbf6d88767a0c5e1dd36c6ee26cefd WatchSource:0}: Error finding container 7b1028c3af87d55375d3daa25d4f253bcbbbf6d88767a0c5e1dd36c6ee26cefd: Status 404 returned error can't find the container with id 7b1028c3af87d55375d3daa25d4f253bcbbbf6d88767a0c5e1dd36c6ee26cefd Oct 30 21:57:09.284380 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:09.284351 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-fc7c456c9-9z74t" event={"ID":"e21a1a66-81c3-4e3f-a37b-5d2c6d799d2e","Type":"ContainerStarted","Data":"7b1028c3af87d55375d3daa25d4f253bcbbbf6d88767a0c5e1dd36c6ee26cefd"} Oct 30 21:57:10.291392 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:10.291353 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-fc7c456c9-9z74t" event={"ID":"e21a1a66-81c3-4e3f-a37b-5d2c6d799d2e","Type":"ContainerStarted","Data":"2d04741a382953db9e90db4a9563579b4bf2e10fa394cd799eee2e701cfab874"} Oct 30 21:57:10.291911 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:10.291500 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-fc7c456c9-9z74t" Oct 30 21:57:10.407338 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:10.407286 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-fc7c456c9-9z74t" podStartSLOduration=2.603325545 podStartE2EDuration="3.407272449s" totalImagesPullingTime="803.946904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:10.407045536 +0000 UTC m=+763.507477268" watchObservedRunningTime="2026-10-30 21:57:10.407272449 +0000 UTC m=+763.507704182" Oct 30 21:57:15.264283 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:15.264251 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-dbz8f" Oct 30 21:57:16.269348 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:16.269313 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qlsfm" Oct 30 21:57:21.298867 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:21.298839 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-fc7c456c9-9z74t" Oct 30 21:57:21.467730 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:21.467700 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-z5wtt"] Oct 30 21:57:21.467941 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:21.467914 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-66dd76bf94-z5wtt" podUID="da0597f7-5b3f-4707-b604-28585e478b00" containerName="manager" containerID="cri-o://5e6cd86e624c6706e332bf59675167f5743c45fa712f0e0ad7eac9ea8b0065d5" gracePeriod=30 Oct 30 21:57:21.713744 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:21.713714 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-z5wtt" Oct 30 21:57:21.742678 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:21.742654 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da0597f7-5b3f-4707-b604-28585e478b00-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0597f7-5b3f-4707-b604-28585e478b00-metrics-certs\") pod \"da0597f7-5b3f-4707-b604-28585e478b00\" (UID: \"da0597f7-5b3f-4707-b604-28585e478b00\") " Oct 30 21:57:21.742801 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:21.742699 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0597f7-5b3f-4707-b604-28585e478b00-kube-api-access-6g89n\" (UniqueName: \"kubernetes.io/projected/da0597f7-5b3f-4707-b604-28585e478b00-kube-api-access-6g89n\") pod \"da0597f7-5b3f-4707-b604-28585e478b00\" (UID: \"da0597f7-5b3f-4707-b604-28585e478b00\") " Oct 30 21:57:21.744772 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:21.744744 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da0597f7-5b3f-4707-b604-28585e478b00-metrics-certs" pod "da0597f7-5b3f-4707-b604-28585e478b00" (UID: "da0597f7-5b3f-4707-b604-28585e478b00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:21.744893 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:21.744843 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0597f7-5b3f-4707-b604-28585e478b00-kube-api-access-6g89n" pod "da0597f7-5b3f-4707-b604-28585e478b00" (UID: "da0597f7-5b3f-4707-b604-28585e478b00"). InnerVolumeSpecName "kube-api-access-6g89n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:21.843576 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:21.843520 4335 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0597f7-5b3f-4707-b604-28585e478b00-metrics-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:21.843576 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:21.843541 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g89n\" (UniqueName: \"kubernetes.io/projected/da0597f7-5b3f-4707-b604-28585e478b00-kube-api-access-6g89n\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:22.368043 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:22.368003 4335 generic.go:356] "Generic (PLEG): container finished" podID="da0597f7-5b3f-4707-b604-28585e478b00" containerID="5e6cd86e624c6706e332bf59675167f5743c45fa712f0e0ad7eac9ea8b0065d5" exitCode=0 Oct 30 21:57:22.368487 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:22.368073 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-z5wtt" Oct 30 21:57:22.368487 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:22.368094 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-z5wtt" event={"ID":"da0597f7-5b3f-4707-b604-28585e478b00","Type":"ContainerDied","Data":"5e6cd86e624c6706e332bf59675167f5743c45fa712f0e0ad7eac9ea8b0065d5"} Oct 30 21:57:22.368487 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:22.368184 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-z5wtt" event={"ID":"da0597f7-5b3f-4707-b604-28585e478b00","Type":"ContainerDied","Data":"324e8711e0877c5ec665892aabe98d6d531908d65b4c77707fedbbb69af2cd8e"} Oct 30 21:57:22.368487 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:22.368217 4335 scope.go:122] "RemoveContainer" containerID="5e6cd86e624c6706e332bf59675167f5743c45fa712f0e0ad7eac9ea8b0065d5" Oct 30 21:57:22.379080 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:22.379062 4335 scope.go:122] "RemoveContainer" containerID="5e6cd86e624c6706e332bf59675167f5743c45fa712f0e0ad7eac9ea8b0065d5" Oct 30 21:57:22.379340 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:57:22.379321 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e6cd86e624c6706e332bf59675167f5743c45fa712f0e0ad7eac9ea8b0065d5\": container with ID starting with 5e6cd86e624c6706e332bf59675167f5743c45fa712f0e0ad7eac9ea8b0065d5 not found: ID does not exist" containerID="5e6cd86e624c6706e332bf59675167f5743c45fa712f0e0ad7eac9ea8b0065d5" Oct 30 21:57:22.379392 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:22.379349 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e6cd86e624c6706e332bf59675167f5743c45fa712f0e0ad7eac9ea8b0065d5"} err="failed to get container status \"5e6cd86e624c6706e332bf59675167f5743c45fa712f0e0ad7eac9ea8b0065d5\": rpc error: code = NotFound desc = could not find container \"5e6cd86e624c6706e332bf59675167f5743c45fa712f0e0ad7eac9ea8b0065d5\": container with ID starting with 5e6cd86e624c6706e332bf59675167f5743c45fa712f0e0ad7eac9ea8b0065d5 not found: ID does not exist" Oct 30 21:57:22.532931 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:22.532908 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-z5wtt"] Oct 30 21:57:22.582230 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:22.582210 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-z5wtt"] Oct 30 21:57:23.056934 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:23.056912 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da0597f7-5b3f-4707-b604-28585e478b00" path="/var/lib/kubelet/pods/da0597f7-5b3f-4707-b604-28585e478b00/volumes" Oct 30 21:57:25.320488 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:57:25.320455 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:57:26.895177 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.895146 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf"] Oct 30 21:57:26.895589 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.895571 4335 cpu_manager.go:414] "containerMap: removing container" podUID="da0597f7-5b3f-4707-b604-28585e478b00" containerName="manager" Oct 30 21:57:26.895589 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.895583 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0597f7-5b3f-4707-b604-28585e478b00" containerName="manager" Oct 30 21:57:26.895686 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.895674 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0597f7-5b3f-4707-b604-28585e478b00" containerName="manager" Oct 30 21:57:26.900264 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.900245 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:26.944450 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.944429 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c"] Oct 30 21:57:26.947767 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.947753 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:26.979760 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.979723 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:26.979854 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.979765 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:26.979854 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.979782 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:26.979931 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.979860 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec3aebd-024a-420b-a207-9ca1d8982d29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:26.979931 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.979919 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnjzk\" (UniqueName: \"kubernetes.io/projected/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-kube-api-access-pnjzk\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:26.980006 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.979946 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:26.980006 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.979968 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:26.980006 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.979986 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:26.980110 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.980056 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:26.980110 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.980072 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:26.980110 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.980091 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:26.980110 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.980107 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:26.980255 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.980162 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft8xr\" (UniqueName: \"kubernetes.io/projected/8ec3aebd-024a-420b-a207-9ca1d8982d29-kube-api-access-ft8xr\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:26.980255 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.980185 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:26.980255 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.980213 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:26.980255 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.980235 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:26.980393 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.980268 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:26.980393 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:26.980288 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.044320 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.044293 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c"] Oct 30 21:57:27.081362 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.081330 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.081470 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.081365 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.081470 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.081397 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec3aebd-024a-420b-a207-9ca1d8982d29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.081470 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.081422 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnjzk\" (UniqueName: \"kubernetes.io/projected/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-kube-api-access-pnjzk\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.081470 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.081450 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.081724 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.081483 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.081724 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.081668 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.081832 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.081728 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.081832 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.081739 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.081832 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.081759 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.081832 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.081790 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.081832 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.081808 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.082083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.081834 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft8xr\" (UniqueName: \"kubernetes.io/projected/8ec3aebd-024a-420b-a207-9ca1d8982d29-kube-api-access-ft8xr\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.082083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.081866 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.082083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.081918 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.082083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.081944 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.082083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.081992 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.082083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.082025 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.082083 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.082058 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.082450 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.082085 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.082450 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.082135 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.082450 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.082213 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.082450 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.082389 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec3aebd-024a-420b-a207-9ca1d8982d29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.082667 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.082516 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.082729 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.082671 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.082729 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.082682 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.082996 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.082960 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.083125 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.082965 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.083914 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.083887 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.084263 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.084245 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.084521 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.084502 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.084927 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.084907 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.112185 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.112164 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf"] Oct 30 21:57:27.145724 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.145653 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.146161 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.146137 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.146270 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.146187 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnjzk\" (UniqueName: \"kubernetes.io/projected/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-kube-api-access-pnjzk\") pod \"maas-default-gateway-openshift-default-6cc458d887-w8jkf\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.146367 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.146346 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft8xr\" (UniqueName: \"kubernetes.io/projected/8ec3aebd-024a-420b-a207-9ca1d8982d29-kube-api-access-ft8xr\") pod \"maas-default-gateway-openshift-default-6cc458d887-gvj5c\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.213533 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.213514 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:27.259687 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.259658 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:27.642927 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.642905 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf"] Oct 30 21:57:27.643283 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:57:27.643262 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f1ef3d_7992_4f3c_8064_3f3c23bb6481.slice/crio-84ab5b6f0086cc2306296d9532ab2d39b4882818b8881d6c27d7041215c5d6d1 WatchSource:0}: Error finding container 84ab5b6f0086cc2306296d9532ab2d39b4882818b8881d6c27d7041215c5d6d1: Status 404 returned error can't find the container with id 84ab5b6f0086cc2306296d9532ab2d39b4882818b8881d6c27d7041215c5d6d1 Oct 30 21:57:27.645196 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.645168 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:57:27.645259 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.645227 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:57:27.645259 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.645255 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:57:27.693056 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.693036 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c"] Oct 30 21:57:27.695013 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:57:27.694992 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec3aebd_024a_420b_a207_9ca1d8982d29.slice/crio-24b9d06d6e42de4b91da04fae617c38cefeb58f6b343e3ea4998d63370d37043 WatchSource:0}: Error finding container 24b9d06d6e42de4b91da04fae617c38cefeb58f6b343e3ea4998d63370d37043: Status 404 returned error can't find the container with id 24b9d06d6e42de4b91da04fae617c38cefeb58f6b343e3ea4998d63370d37043 Oct 30 21:57:27.697127 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.697096 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:57:27.697214 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.697165 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:57:27.697214 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:27.697195 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:57:28.447226 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:28.447187 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" event={"ID":"8ec3aebd-024a-420b-a207-9ca1d8982d29","Type":"ContainerStarted","Data":"1932847dd06d0a4fb3029dd27f518af3d99aab9c4630648af2b7b33afad16edd"} Oct 30 21:57:28.447796 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:28.447263 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" event={"ID":"8ec3aebd-024a-420b-a207-9ca1d8982d29","Type":"ContainerStarted","Data":"24b9d06d6e42de4b91da04fae617c38cefeb58f6b343e3ea4998d63370d37043"} Oct 30 21:57:28.450980 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:28.450954 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" event={"ID":"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481","Type":"ContainerStarted","Data":"0e5b2586ce5c1bf9d8a0697c2f9419fc88598a997f998c3ea14dfd1383aed9c1"} Oct 30 21:57:28.451174 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:28.451154 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" event={"ID":"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481","Type":"ContainerStarted","Data":"84ab5b6f0086cc2306296d9532ab2d39b4882818b8881d6c27d7041215c5d6d1"} Oct 30 21:57:28.643641 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:28.625366 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" podStartSLOduration=2.625346941 podStartE2EDuration="2.625346941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:28.622075369 +0000 UTC m=+781.722507102" watchObservedRunningTime="2026-10-30 21:57:28.625346941 +0000 UTC m=+781.725778676" Oct 30 21:57:28.751794 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:28.751665 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" podStartSLOduration=2.751644063 podStartE2EDuration="2.751644063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:28.746988674 +0000 UTC m=+781.847420426" watchObservedRunningTime="2026-10-30 21:57:28.751644063 +0000 UTC m=+781.852075795" Oct 30 21:57:29.218641 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:29.217390 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:29.218641 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:29.217578 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w8jkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:57:29.218641 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:29.217657 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" podUID="d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:57:29.259870 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:29.259835 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:29.261022 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:29.260998 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-gvj5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:57:29.261127 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:29.261043 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" podUID="8ec3aebd-024a-420b-a207-9ca1d8982d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 21:57:30.218963 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:30.218925 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w8jkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:57:30.219645 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:30.219588 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" podUID="d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:57:30.260134 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:30.260104 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-gvj5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:57:30.260262 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:30.260150 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" podUID="8ec3aebd-024a-420b-a207-9ca1d8982d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 21:57:31.214027 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:31.213988 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w8jkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:57:31.214333 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:31.214302 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" podUID="d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:57:31.260853 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:31.260823 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-gvj5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:57:31.261298 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:31.261270 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" podUID="8ec3aebd-024a-420b-a207-9ca1d8982d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 21:57:32.214047 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:32.214013 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w8jkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:57:32.214438 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:32.214408 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" podUID="d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:57:32.260641 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:32.260593 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-gvj5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:57:32.260823 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:32.260660 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" podUID="8ec3aebd-024a-420b-a207-9ca1d8982d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 21:57:33.219638 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:33.219022 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w8jkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:57:33.219638 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:33.219079 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" podUID="d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:57:33.260020 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:33.259985 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-gvj5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:57:33.260180 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:33.260041 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" podUID="8ec3aebd-024a-420b-a207-9ca1d8982d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 21:57:34.214714 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:34.214680 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w8jkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:57:34.214879 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:34.214745 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" podUID="d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:57:34.260777 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:34.260749 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-gvj5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:57:34.261073 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:34.260804 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" podUID="8ec3aebd-024a-420b-a207-9ca1d8982d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 21:57:35.975723 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:35.975688 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:35.976422 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:35.976401 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:35.977297 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:35.977270 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 21:57:35.991189 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:35.991166 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:35.991690 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:35.991669 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:35.992370 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:57:35.992347 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 21:57:55.320062 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:57:55.320026 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:58:25.319893 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:58:25.319858 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:58:26.239519 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.239489 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4"] Oct 30 21:58:26.242122 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.242102 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.303956 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.303931 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4"] Oct 30 21:58:26.326126 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.326102 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-workload-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.326562 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.326129 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-podinfo\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.326562 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.326154 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-envoy\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.326562 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.326297 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istiod-ca-cert\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.326562 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.326359 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2ghc\" (UniqueName: \"kubernetes.io/projected/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-kube-api-access-z2ghc\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.326562 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.326397 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-workload-certs\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.326562 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.326469 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-data\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.326562 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.326534 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-credential-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.326562 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.326553 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-token\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.392575 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.392553 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b2f165-openshift-default-dockercfg-5fv84\"" Oct 30 21:58:26.427145 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.427117 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-credential-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.427145 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.427145 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-token\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.427307 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.427199 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-workload-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.427307 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.427217 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-podinfo\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.427307 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.427244 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-envoy\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.427307 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.427291 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istiod-ca-cert\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.427504 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.427320 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2ghc\" (UniqueName: \"kubernetes.io/projected/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-kube-api-access-z2ghc\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.427504 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.427351 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-workload-certs\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.427504 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.427418 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-data\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.427699 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.427607 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-workload-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.427808 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.427791 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-data\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.427989 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.427944 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-credential-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.427989 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.427949 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-workload-certs\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.428205 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.428146 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istiod-ca-cert\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.429647 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.429598 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-envoy\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.429949 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.429931 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-podinfo\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.492304 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.492197 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-token\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.493875 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.493851 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2ghc\" (UniqueName: \"kubernetes.io/projected/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-kube-api-access-z2ghc\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.554087 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.554051 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:26.858149 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.858116 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92"] Oct 30 21:58:26.861603 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.861581 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:26.913535 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.913511 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-a7b06b-openshift-default-dockercfg-2f8cv\"" Oct 30 21:58:26.932939 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.932911 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92"] Oct 30 21:58:26.933243 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.933211 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-data\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:26.933351 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.933279 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-podinfo\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:26.933400 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.933363 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkgdz\" (UniqueName: \"kubernetes.io/projected/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-kube-api-access-xkgdz\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:26.933437 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.933416 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-workload-socket\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:26.933504 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.933486 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-token\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:26.933539 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.933527 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istiod-ca-cert\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:26.933573 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.933551 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-credential-socket\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:26.933676 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.933659 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-workload-certs\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:26.933720 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:26.933706 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-envoy\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.034763 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.034725 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-podinfo\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.035199 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.034787 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkgdz\" (UniqueName: \"kubernetes.io/projected/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-kube-api-access-xkgdz\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.035199 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.034869 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-workload-socket\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.035199 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.034913 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-token\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.035199 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.034954 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istiod-ca-cert\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.035199 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.034979 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-credential-socket\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.035199 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.035004 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-workload-certs\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.035199 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.035047 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-envoy\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.035199 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.035096 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-data\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.035972 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.035945 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-workload-socket\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.036041 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.035994 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-data\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.036536 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.036502 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istiod-ca-cert\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.036674 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.036513 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-workload-certs\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.036776 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.036755 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-credential-socket\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.039029 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.039005 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-envoy\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.039163 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.039144 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-podinfo\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.039510 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.039486 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4"] Oct 30 21:58:27.040698 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:58:27.040670 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8ee860_7c1a_43f1_8e21_634a5bc0dee1.slice/crio-9074ab260ad5c3b7e55dade916d6ce00c4b1420401b7d3872057c0e3b6119ed8 WatchSource:0}: Error finding container 9074ab260ad5c3b7e55dade916d6ce00c4b1420401b7d3872057c0e3b6119ed8: Status 404 returned error can't find the container with id 9074ab260ad5c3b7e55dade916d6ce00c4b1420401b7d3872057c0e3b6119ed8 Oct 30 21:58:27.042807 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.042775 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:58:27.042881 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.042837 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:58:27.042881 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.042867 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:58:27.096591 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.096564 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-token\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.096954 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.096929 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkgdz\" (UniqueName: \"kubernetes.io/projected/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-kube-api-access-xkgdz\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.224760 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.224692 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-a7b06b-openshift-default-dockercfg-2f8cv\"" Oct 30 21:58:27.233962 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.233934 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:27.677063 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.677031 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92"] Oct 30 21:58:27.677401 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:58:27.677065 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2eb14d_86a7_4cdb_919c_9a5d1296d5d4.slice/crio-741fe34af64372362e02ee4ed5de573a100ecb3b10f199c60b1f56676f8c4825 WatchSource:0}: Error finding container 741fe34af64372362e02ee4ed5de573a100ecb3b10f199c60b1f56676f8c4825: Status 404 returned error can't find the container with id 741fe34af64372362e02ee4ed5de573a100ecb3b10f199c60b1f56676f8c4825 Oct 30 21:58:27.679582 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.679550 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:58:27.679713 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.679647 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:58:27.679713 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.679693 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:58:27.834606 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.834459 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" event={"ID":"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4","Type":"ContainerStarted","Data":"06b9a0c7bd9b0499ff12b761840ec9569aab185e82eabfa8d9e23ab9c1feb1f9"} Oct 30 21:58:27.834606 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.834525 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" event={"ID":"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4","Type":"ContainerStarted","Data":"741fe34af64372362e02ee4ed5de573a100ecb3b10f199c60b1f56676f8c4825"} Oct 30 21:58:27.836584 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.836560 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" event={"ID":"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1","Type":"ContainerStarted","Data":"03850171021aacd90821ddd621b0e1ea020bcd503e73f6ffd955d922de2635b8"} Oct 30 21:58:27.836755 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.836640 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" event={"ID":"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1","Type":"ContainerStarted","Data":"9074ab260ad5c3b7e55dade916d6ce00c4b1420401b7d3872057c0e3b6119ed8"} Oct 30 21:58:27.957315 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:27.957203 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" podStartSLOduration=1.957184425 podStartE2EDuration="1.957184425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:27.954884681 +0000 UTC m=+841.055316412" watchObservedRunningTime="2026-10-30 21:58:27.957184425 +0000 UTC m=+841.057616158" Oct 30 21:58:28.087684 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:28.087602 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" podStartSLOduration=2.087581551 podStartE2EDuration="2.087581551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:28.074433875 +0000 UTC m=+841.174865620" watchObservedRunningTime="2026-10-30 21:58:28.087581551 +0000 UTC m=+841.188013284" Oct 30 21:58:28.234450 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:28.234367 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:28.239610 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:28.239584 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:28.554680 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:28.554654 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:28.559381 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:28.559361 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:28.842281 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:28.842205 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:28.842281 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:28.842259 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:28.842930 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:28.842911 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:58:28.843126 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:28.843110 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:35.630414 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.630382 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j"] Oct 30 21:58:35.633139 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.633120 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" Oct 30 21:58:35.687936 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.687909 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-7df73b-dockercfg-rq4sk\"" Oct 30 21:58:35.718260 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.718227 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85380bc-9459-4ef3-94c1-663683361092-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j\" (UID: \"f85380bc-9459-4ef3-94c1-663683361092\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" Oct 30 21:58:35.718396 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.718346 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2qpp\" (UniqueName: \"kubernetes.io/projected/f85380bc-9459-4ef3-94c1-663683361092-kube-api-access-j2qpp\") pod \"payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j\" (UID: \"f85380bc-9459-4ef3-94c1-663683361092\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" Oct 30 21:58:35.737907 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.737875 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-7df73b\"" Oct 30 21:58:35.740959 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.740922 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j"] Oct 30 21:58:35.797206 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.797172 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn"] Oct 30 21:58:35.800603 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.800570 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" Oct 30 21:58:35.819961 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.819211 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2qpp\" (UniqueName: \"kubernetes.io/projected/f85380bc-9459-4ef3-94c1-663683361092-kube-api-access-j2qpp\") pod \"payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j\" (UID: \"f85380bc-9459-4ef3-94c1-663683361092\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" Oct 30 21:58:35.819961 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.819346 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85380bc-9459-4ef3-94c1-663683361092-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j\" (UID: \"f85380bc-9459-4ef3-94c1-663683361092\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" Oct 30 21:58:35.820257 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.820232 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85380bc-9459-4ef3-94c1-663683361092-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j\" (UID: \"f85380bc-9459-4ef3-94c1-663683361092\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" Oct 30 21:58:35.851841 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.851814 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn"] Oct 30 21:58:35.881600 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.881539 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2qpp\" (UniqueName: \"kubernetes.io/projected/f85380bc-9459-4ef3-94c1-663683361092-kube-api-access-j2qpp\") pod \"payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j\" (UID: \"f85380bc-9459-4ef3-94c1-663683361092\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" Oct 30 21:58:35.902402 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.902375 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996"] Oct 30 21:58:35.905455 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.905435 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:35.921078 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.921053 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7601c3-d83d-4d7f-b5d3-7528cba61d67-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn\" (UID: \"8a7601c3-d83d-4d7f-b5d3-7528cba61d67\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" Oct 30 21:58:35.921171 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.921115 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsnbt\" (UniqueName: \"kubernetes.io/projected/8a7601c3-d83d-4d7f-b5d3-7528cba61d67-kube-api-access-lsnbt\") pod \"payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn\" (UID: \"8a7601c3-d83d-4d7f-b5d3-7528cba61d67\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" Oct 30 21:58:35.969827 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.969777 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-bf7cbf-openshift-default-dockercfg-7rlzz\"" Oct 30 21:58:35.988527 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.987847 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" Oct 30 21:58:35.988527 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:35.988205 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996"] Oct 30 21:58:36.041654 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.040147 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-credential-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.041654 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.040210 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-workload-certs\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.041654 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.040269 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-envoy\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.041654 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.040322 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-podinfo\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.041654 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.040352 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istiod-ca-cert\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.041654 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.040396 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k58tk\" (UniqueName: \"kubernetes.io/projected/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-kube-api-access-k58tk\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.041654 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.040423 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-data\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.041654 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.040477 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-workload-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.041654 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.040551 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7601c3-d83d-4d7f-b5d3-7528cba61d67-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn\" (UID: \"8a7601c3-d83d-4d7f-b5d3-7528cba61d67\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" Oct 30 21:58:36.041654 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.040582 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-token\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.041654 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.041213 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7601c3-d83d-4d7f-b5d3-7528cba61d67-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn\" (UID: \"8a7601c3-d83d-4d7f-b5d3-7528cba61d67\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" Oct 30 21:58:36.041654 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.041337 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsnbt\" (UniqueName: \"kubernetes.io/projected/8a7601c3-d83d-4d7f-b5d3-7528cba61d67-kube-api-access-lsnbt\") pod \"payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn\" (UID: \"8a7601c3-d83d-4d7f-b5d3-7528cba61d67\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" Oct 30 21:58:36.105323 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.105154 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsnbt\" (UniqueName: \"kubernetes.io/projected/8a7601c3-d83d-4d7f-b5d3-7528cba61d67-kube-api-access-lsnbt\") pod \"payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn\" (UID: \"8a7601c3-d83d-4d7f-b5d3-7528cba61d67\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" Oct 30 21:58:36.124883 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.124798 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" Oct 30 21:58:36.142586 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.142559 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-envoy\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.142745 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.142637 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-podinfo\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.142745 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.142669 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istiod-ca-cert\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.142745 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.142708 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k58tk\" (UniqueName: \"kubernetes.io/projected/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-kube-api-access-k58tk\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.142745 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.142735 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-data\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.142957 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.142785 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-workload-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.142957 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.142817 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-token\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.142957 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.142952 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-credential-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.143112 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.142982 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-workload-certs\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.144450 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.143458 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-data\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.144450 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.143856 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-workload-certs\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.144450 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.143921 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-workload-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.144450 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.144094 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-credential-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.144450 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.144216 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istiod-ca-cert\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.145995 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.145969 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-envoy\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.147271 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.147250 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-podinfo\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.209711 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.209679 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-token\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.210773 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.210746 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k58tk\" (UniqueName: \"kubernetes.io/projected/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-kube-api-access-k58tk\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.219760 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.219726 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:36.416192 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.416159 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j"] Oct 30 21:58:36.417672 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:58:36.417645 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85380bc_9459_4ef3_94c1_663683361092.slice/crio-a0f87780893e6da22a73b64609ac73565a8958a36d4c305a4c2740856c0ebf16 WatchSource:0}: Error finding container a0f87780893e6da22a73b64609ac73565a8958a36d4c305a4c2740856c0ebf16: Status 404 returned error can't find the container with id a0f87780893e6da22a73b64609ac73565a8958a36d4c305a4c2740856c0ebf16 Oct 30 21:58:36.501122 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.501095 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn"] Oct 30 21:58:36.501961 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:58:36.501926 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7601c3_d83d_4d7f_b5d3_7528cba61d67.slice/crio-cfb567cbda392ac18caa1ed91e2df3d1de12a36cc7b500964177b73c84e00209 WatchSource:0}: Error finding container cfb567cbda392ac18caa1ed91e2df3d1de12a36cc7b500964177b73c84e00209: Status 404 returned error can't find the container with id cfb567cbda392ac18caa1ed91e2df3d1de12a36cc7b500964177b73c84e00209 Oct 30 21:58:36.609185 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.609161 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996"] Oct 30 21:58:36.610016 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:58:36.609994 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7b76ed_f54c_4ff1_930a_c5f183bc72ec.slice/crio-ce45e0851ece194cd868bf3bc60915f94dcbed0a5451c98026a614fdf600aed5 WatchSource:0}: Error finding container ce45e0851ece194cd868bf3bc60915f94dcbed0a5451c98026a614fdf600aed5: Status 404 returned error can't find the container with id ce45e0851ece194cd868bf3bc60915f94dcbed0a5451c98026a614fdf600aed5 Oct 30 21:58:36.612143 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.612117 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:58:36.612204 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.612183 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:58:36.612244 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.612215 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:58:36.895815 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.895778 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" event={"ID":"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec","Type":"ContainerStarted","Data":"1b75afff3530fae1c95aa5cd6755bd7ddbdf89a857a28a58ae17a03cb2984e85"} Oct 30 21:58:36.896250 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.895840 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" event={"ID":"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec","Type":"ContainerStarted","Data":"ce45e0851ece194cd868bf3bc60915f94dcbed0a5451c98026a614fdf600aed5"} Oct 30 21:58:36.897559 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.897524 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" event={"ID":"f85380bc-9459-4ef3-94c1-663683361092","Type":"ContainerStarted","Data":"6c22a1475ab2a012cdf840909d2814a8ab103f4c6afb739adfc722cdb120ecba"} Oct 30 21:58:36.897709 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.897578 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" event={"ID":"f85380bc-9459-4ef3-94c1-663683361092","Type":"ContainerStarted","Data":"a0f87780893e6da22a73b64609ac73565a8958a36d4c305a4c2740856c0ebf16"} Oct 30 21:58:36.897709 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.897641 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" Oct 30 21:58:36.899137 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.899107 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" event={"ID":"8a7601c3-d83d-4d7f-b5d3-7528cba61d67","Type":"ContainerStarted","Data":"db3836b80dd561fdf8585ab351e6e3a3d9211908021d520c7af2c91b8e8b740a"} Oct 30 21:58:36.899251 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.899156 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" event={"ID":"8a7601c3-d83d-4d7f-b5d3-7528cba61d67","Type":"ContainerStarted","Data":"cfb567cbda392ac18caa1ed91e2df3d1de12a36cc7b500964177b73c84e00209"} Oct 30 21:58:36.899251 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:36.899205 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" Oct 30 21:58:37.022599 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:37.022493 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" podStartSLOduration=2.022474546 podStartE2EDuration="2.022474546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:37.020466924 +0000 UTC m=+850.120898660" watchObservedRunningTime="2026-10-30 21:58:37.022474546 +0000 UTC m=+850.122906281" Oct 30 21:58:37.155513 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:37.155466 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" podStartSLOduration=2.155453584 podStartE2EDuration="2.155453584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:37.152900304 +0000 UTC m=+850.253332036" watchObservedRunningTime="2026-10-30 21:58:37.155453584 +0000 UTC m=+850.255885364" Oct 30 21:58:37.220663 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:37.220602 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:37.226337 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:37.226300 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:37.274045 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:37.273957 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" podStartSLOduration=2.2739423690000002 podStartE2EDuration="2.273942369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:37.27212426 +0000 UTC m=+850.372555991" watchObservedRunningTime="2026-10-30 21:58:37.273942369 +0000 UTC m=+850.374374100" Oct 30 21:58:37.906806 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:37.906775 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:37.907824 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:37.907800 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:58:38.174306 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.174195 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q"] Oct 30 21:58:38.176811 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.176796 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" Oct 30 21:58:38.235311 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.235279 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-b2f165\"" Oct 30 21:58:38.235453 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.235349 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-b2f165-dockercfg-fq8bf\"" Oct 30 21:58:38.238665 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.238641 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q"] Oct 30 21:58:38.277651 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.277459 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbb21e7-621e-4490-bc0b-f31ee5b34404-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q\" (UID: \"9bbb21e7-621e-4490-bc0b-f31ee5b34404\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" Oct 30 21:58:38.277850 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.277736 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vjv7\" (UniqueName: \"kubernetes.io/projected/9bbb21e7-621e-4490-bc0b-f31ee5b34404-kube-api-access-2vjv7\") pod \"payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q\" (UID: \"9bbb21e7-621e-4490-bc0b-f31ee5b34404\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" Oct 30 21:58:38.321335 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.321305 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj"] Oct 30 21:58:38.323736 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.323721 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" Oct 30 21:58:38.379084 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.378997 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbb21e7-621e-4490-bc0b-f31ee5b34404-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q\" (UID: \"9bbb21e7-621e-4490-bc0b-f31ee5b34404\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" Oct 30 21:58:38.379225 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.379124 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vjv7\" (UniqueName: \"kubernetes.io/projected/9bbb21e7-621e-4490-bc0b-f31ee5b34404-kube-api-access-2vjv7\") pod \"payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q\" (UID: \"9bbb21e7-621e-4490-bc0b-f31ee5b34404\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" Oct 30 21:58:38.380314 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.380286 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbb21e7-621e-4490-bc0b-f31ee5b34404-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q\" (UID: \"9bbb21e7-621e-4490-bc0b-f31ee5b34404\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" Oct 30 21:58:38.383414 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.383389 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj"] Oct 30 21:58:38.441865 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.441799 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vjv7\" (UniqueName: \"kubernetes.io/projected/9bbb21e7-621e-4490-bc0b-f31ee5b34404-kube-api-access-2vjv7\") pod \"payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q\" (UID: \"9bbb21e7-621e-4490-bc0b-f31ee5b34404\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" Oct 30 21:58:38.480433 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.480405 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1f40ea-c3a3-4522-b526-fb3aad6183fa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj\" (UID: \"ca1f40ea-c3a3-4522-b526-fb3aad6183fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" Oct 30 21:58:38.480549 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.480451 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tzw8\" (UniqueName: \"kubernetes.io/projected/ca1f40ea-c3a3-4522-b526-fb3aad6183fa-kube-api-access-6tzw8\") pod \"payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj\" (UID: \"ca1f40ea-c3a3-4522-b526-fb3aad6183fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" Oct 30 21:58:38.490302 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.490281 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" Oct 30 21:58:38.581868 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.581838 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1f40ea-c3a3-4522-b526-fb3aad6183fa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj\" (UID: \"ca1f40ea-c3a3-4522-b526-fb3aad6183fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" Oct 30 21:58:38.581996 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.581888 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tzw8\" (UniqueName: \"kubernetes.io/projected/ca1f40ea-c3a3-4522-b526-fb3aad6183fa-kube-api-access-6tzw8\") pod \"payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj\" (UID: \"ca1f40ea-c3a3-4522-b526-fb3aad6183fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" Oct 30 21:58:38.582767 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.582731 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1f40ea-c3a3-4522-b526-fb3aad6183fa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj\" (UID: \"ca1f40ea-c3a3-4522-b526-fb3aad6183fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" Oct 30 21:58:38.643656 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.643609 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tzw8\" (UniqueName: \"kubernetes.io/projected/ca1f40ea-c3a3-4522-b526-fb3aad6183fa-kube-api-access-6tzw8\") pod \"payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj\" (UID: \"ca1f40ea-c3a3-4522-b526-fb3aad6183fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" Oct 30 21:58:38.872823 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.872792 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q"] Oct 30 21:58:38.873164 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:58:38.873143 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbb21e7_621e_4490_bc0b_f31ee5b34404.slice/crio-a2e17639cc3ab420469a33ba5cb83cd7142412e7b8a08b7915107389ab215914 WatchSource:0}: Error finding container a2e17639cc3ab420469a33ba5cb83cd7142412e7b8a08b7915107389ab215914: Status 404 returned error can't find the container with id a2e17639cc3ab420469a33ba5cb83cd7142412e7b8a08b7915107389ab215914 Oct 30 21:58:38.921452 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.921416 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" event={"ID":"9bbb21e7-621e-4490-bc0b-f31ee5b34404","Type":"ContainerStarted","Data":"a2e17639cc3ab420469a33ba5cb83cd7142412e7b8a08b7915107389ab215914"} Oct 30 21:58:38.935695 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:38.935672 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" Oct 30 21:58:39.313253 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:39.313229 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj"] Oct 30 21:58:39.313759 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:58:39.313734 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1f40ea_c3a3_4522_b526_fb3aad6183fa.slice/crio-b1289d1f60574c2665c10d688b480a69c5a464e46cb0c1ee74071a98350a1e24 WatchSource:0}: Error finding container b1289d1f60574c2665c10d688b480a69c5a464e46cb0c1ee74071a98350a1e24: Status 404 returned error can't find the container with id b1289d1f60574c2665c10d688b480a69c5a464e46cb0c1ee74071a98350a1e24 Oct 30 21:58:39.930875 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:39.930847 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" event={"ID":"ca1f40ea-c3a3-4522-b526-fb3aad6183fa","Type":"ContainerStarted","Data":"fde7e5a9aed94d157063a00d2e6bbe997f6916935068b72a84f500aa331eef6b"} Oct 30 21:58:39.931345 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:39.930894 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" event={"ID":"ca1f40ea-c3a3-4522-b526-fb3aad6183fa","Type":"ContainerStarted","Data":"b1289d1f60574c2665c10d688b480a69c5a464e46cb0c1ee74071a98350a1e24"} Oct 30 21:58:39.931345 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:39.931019 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" Oct 30 21:58:39.932505 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:39.932480 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" event={"ID":"9bbb21e7-621e-4490-bc0b-f31ee5b34404","Type":"ContainerStarted","Data":"54f53254f640eb49e4801bdeb5dc98c0b1647485b077e8648873a3938eee7721"} Oct 30 21:58:40.049295 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:40.049242 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" podStartSLOduration=2.049228443 podStartE2EDuration="2.049228443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:40.047735217 +0000 UTC m=+853.148166950" watchObservedRunningTime="2026-10-30 21:58:40.049228443 +0000 UTC m=+853.149660174" Oct 30 21:58:40.167346 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:40.167283 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" podStartSLOduration=2.167263732 podStartE2EDuration="2.167263732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:40.163398182 +0000 UTC m=+853.263829913" watchObservedRunningTime="2026-10-30 21:58:40.167263732 +0000 UTC m=+853.267695461" Oct 30 21:58:40.726733 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:40.726702 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh"] Oct 30 21:58:40.729484 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:40.729463 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" Oct 30 21:58:40.784209 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:40.784178 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-a7b06b\"" Oct 30 21:58:40.788017 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:40.787995 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh"] Oct 30 21:58:40.823640 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:40.820236 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wl8d\" (UniqueName: \"kubernetes.io/projected/0b00cf8d-cd32-402b-8815-0c239365c8da-kube-api-access-5wl8d\") pod \"payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh\" (UID: \"0b00cf8d-cd32-402b-8815-0c239365c8da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" Oct 30 21:58:40.842763 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:40.842727 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b00cf8d-cd32-402b-8815-0c239365c8da-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh\" (UID: \"0b00cf8d-cd32-402b-8815-0c239365c8da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" Oct 30 21:58:40.843261 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:40.843239 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-a7b06b-dockercfg-vdmxf\"" Oct 30 21:58:40.939828 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:40.939788 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" Oct 30 21:58:40.943717 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:40.943684 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b00cf8d-cd32-402b-8815-0c239365c8da-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh\" (UID: \"0b00cf8d-cd32-402b-8815-0c239365c8da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" Oct 30 21:58:40.943887 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:40.943807 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wl8d\" (UniqueName: \"kubernetes.io/projected/0b00cf8d-cd32-402b-8815-0c239365c8da-kube-api-access-5wl8d\") pod \"payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh\" (UID: \"0b00cf8d-cd32-402b-8815-0c239365c8da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" Oct 30 21:58:40.944681 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:40.944660 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b00cf8d-cd32-402b-8815-0c239365c8da-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh\" (UID: \"0b00cf8d-cd32-402b-8815-0c239365c8da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" Oct 30 21:58:41.004795 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:41.004743 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wl8d\" (UniqueName: \"kubernetes.io/projected/0b00cf8d-cd32-402b-8815-0c239365c8da-kube-api-access-5wl8d\") pod \"payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh\" (UID: \"0b00cf8d-cd32-402b-8815-0c239365c8da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" Oct 30 21:58:41.042713 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:41.042692 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" Oct 30 21:58:41.424350 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:41.424317 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh"] Oct 30 21:58:41.425516 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:58:41.425482 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b00cf8d_cd32_402b_8815_0c239365c8da.slice/crio-89c7fe63c2beae7d5ffe2b267627633224525546cc08d3b3e4b3ecb56ef9287c WatchSource:0}: Error finding container 89c7fe63c2beae7d5ffe2b267627633224525546cc08d3b3e4b3ecb56ef9287c: Status 404 returned error can't find the container with id 89c7fe63c2beae7d5ffe2b267627633224525546cc08d3b3e4b3ecb56ef9287c Oct 30 21:58:41.947362 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:41.947331 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" event={"ID":"0b00cf8d-cd32-402b-8815-0c239365c8da","Type":"ContainerStarted","Data":"0c98e66e4b934461163e898748e1ae16e7135859a309098fa95feb09ba51d51c"} Oct 30 21:58:41.947776 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:41.947388 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" event={"ID":"0b00cf8d-cd32-402b-8815-0c239365c8da","Type":"ContainerStarted","Data":"89c7fe63c2beae7d5ffe2b267627633224525546cc08d3b3e4b3ecb56ef9287c"} Oct 30 21:58:41.947776 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:41.947644 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" Oct 30 21:58:43.230509 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.230435 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" podStartSLOduration=3.2304167 podStartE2EDuration="3.2304167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:42.066874161 +0000 UTC m=+855.167305895" watchObservedRunningTime="2026-10-30 21:58:43.2304167 +0000 UTC m=+856.330848445" Oct 30 21:58:43.233278 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.233257 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df"] Oct 30 21:58:43.236951 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.236936 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" Oct 30 21:58:43.266634 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.266591 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff74v\" (UniqueName: \"kubernetes.io/projected/14fc164d-631a-47ae-8772-162c3688aec0-kube-api-access-ff74v\") pod \"payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df\" (UID: \"14fc164d-631a-47ae-8772-162c3688aec0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" Oct 30 21:58:43.266733 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.266651 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fc164d-631a-47ae-8772-162c3688aec0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df\" (UID: \"14fc164d-631a-47ae-8772-162c3688aec0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" Oct 30 21:58:43.290829 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.290809 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-b9fbe2\"" Oct 30 21:58:43.290999 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.290844 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-b9fbe2-dockercfg-97tpd\"" Oct 30 21:58:43.294522 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.294501 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df"] Oct 30 21:58:43.366654 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.366583 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 21:58:43.367873 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.367791 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff74v\" (UniqueName: \"kubernetes.io/projected/14fc164d-631a-47ae-8772-162c3688aec0-kube-api-access-ff74v\") pod \"payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df\" (UID: \"14fc164d-631a-47ae-8772-162c3688aec0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" Oct 30 21:58:43.367873 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.367839 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fc164d-631a-47ae-8772-162c3688aec0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df\" (UID: \"14fc164d-631a-47ae-8772-162c3688aec0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" Oct 30 21:58:43.368545 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.368523 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fc164d-631a-47ae-8772-162c3688aec0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df\" (UID: \"14fc164d-631a-47ae-8772-162c3688aec0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" Oct 30 21:58:43.376926 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.376901 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27"] Oct 30 21:58:43.380829 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.380810 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" Oct 30 21:58:43.430222 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.430198 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff74v\" (UniqueName: \"kubernetes.io/projected/14fc164d-631a-47ae-8772-162c3688aec0-kube-api-access-ff74v\") pod \"payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df\" (UID: \"14fc164d-631a-47ae-8772-162c3688aec0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" Oct 30 21:58:43.438461 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.438437 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27"] Oct 30 21:58:43.469067 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.469032 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32706948-1fdd-41a7-91a0-4b7b7a1db425-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27\" (UID: \"32706948-1fdd-41a7-91a0-4b7b7a1db425\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" Oct 30 21:58:43.475220 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.472352 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k2mx\" (UniqueName: \"kubernetes.io/projected/32706948-1fdd-41a7-91a0-4b7b7a1db425-kube-api-access-4k2mx\") pod \"payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27\" (UID: \"32706948-1fdd-41a7-91a0-4b7b7a1db425\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" Oct 30 21:58:43.551402 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.551372 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" Oct 30 21:58:43.576787 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.576151 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k2mx\" (UniqueName: \"kubernetes.io/projected/32706948-1fdd-41a7-91a0-4b7b7a1db425-kube-api-access-4k2mx\") pod \"payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27\" (UID: \"32706948-1fdd-41a7-91a0-4b7b7a1db425\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" Oct 30 21:58:43.576787 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.576255 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32706948-1fdd-41a7-91a0-4b7b7a1db425-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27\" (UID: \"32706948-1fdd-41a7-91a0-4b7b7a1db425\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" Oct 30 21:58:43.577141 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.577118 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32706948-1fdd-41a7-91a0-4b7b7a1db425-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27\" (UID: \"32706948-1fdd-41a7-91a0-4b7b7a1db425\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" Oct 30 21:58:43.640756 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.640685 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k2mx\" (UniqueName: \"kubernetes.io/projected/32706948-1fdd-41a7-91a0-4b7b7a1db425-kube-api-access-4k2mx\") pod \"payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27\" (UID: \"32706948-1fdd-41a7-91a0-4b7b7a1db425\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" Oct 30 21:58:43.694454 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.694425 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" Oct 30 21:58:43.948499 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.948473 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df"] Oct 30 21:58:43.949297 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:58:43.949272 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14fc164d_631a_47ae_8772_162c3688aec0.slice/crio-8865c28f96cb38dc4f47db3a8425536bdae33aaa104d30687bb4d1aef6f4bf0a WatchSource:0}: Error finding container 8865c28f96cb38dc4f47db3a8425536bdae33aaa104d30687bb4d1aef6f4bf0a: Status 404 returned error can't find the container with id 8865c28f96cb38dc4f47db3a8425536bdae33aaa104d30687bb4d1aef6f4bf0a Oct 30 21:58:43.978925 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:43.978894 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" event={"ID":"14fc164d-631a-47ae-8772-162c3688aec0","Type":"ContainerStarted","Data":"8865c28f96cb38dc4f47db3a8425536bdae33aaa104d30687bb4d1aef6f4bf0a"} Oct 30 21:58:44.093507 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:44.093483 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27"] Oct 30 21:58:44.094185 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:58:44.094163 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32706948_1fdd_41a7_91a0_4b7b7a1db425.slice/crio-91e0e6c686575eb529c09fea0d217d264dfae011bc7664f7061d8570ce3f5e67 WatchSource:0}: Error finding container 91e0e6c686575eb529c09fea0d217d264dfae011bc7664f7061d8570ce3f5e67: Status 404 returned error can't find the container with id 91e0e6c686575eb529c09fea0d217d264dfae011bc7664f7061d8570ce3f5e67 Oct 30 21:58:44.986521 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:44.986483 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" event={"ID":"32706948-1fdd-41a7-91a0-4b7b7a1db425","Type":"ContainerStarted","Data":"cd0e42ecf443a2ac8f6f4b3cb5a0da546dd83bf6b7dc34ca3efa377b718c8858"} Oct 30 21:58:44.987021 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:44.986543 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" event={"ID":"32706948-1fdd-41a7-91a0-4b7b7a1db425","Type":"ContainerStarted","Data":"91e0e6c686575eb529c09fea0d217d264dfae011bc7664f7061d8570ce3f5e67"} Oct 30 21:58:44.987021 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:44.986683 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" Oct 30 21:58:44.988148 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:44.988123 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" event={"ID":"14fc164d-631a-47ae-8772-162c3688aec0","Type":"ContainerStarted","Data":"95e99977e022d1248f3fac90459e9fdbb622df42701f6ec7f4c69c3b42919803"} Oct 30 21:58:44.988276 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:44.988164 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" Oct 30 21:58:45.110669 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:45.110591 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" podStartSLOduration=2.110572673 podStartE2EDuration="2.110572673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:45.106041376 +0000 UTC m=+858.206473108" watchObservedRunningTime="2026-10-30 21:58:45.110572673 +0000 UTC m=+858.211004405" Oct 30 21:58:45.982886 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:45.982820 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" podStartSLOduration=2.982802043 podStartE2EDuration="2.982802043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:45.232476904 +0000 UTC m=+858.332908638" watchObservedRunningTime="2026-10-30 21:58:45.982802043 +0000 UTC m=+859.083233774" Oct 30 21:58:45.985583 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:45.985563 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j"] Oct 30 21:58:45.989575 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:45.989554 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" Oct 30 21:58:46.052345 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:46.052307 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-536975\"" Oct 30 21:58:46.088444 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:46.078769 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j"] Oct 30 21:58:46.097685 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:46.097541 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-536975-dockercfg-bwhtk\"" Oct 30 21:58:46.122542 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:46.122188 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ksg9\" (UniqueName: \"kubernetes.io/projected/f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31-kube-api-access-9ksg9\") pod \"payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j\" (UID: \"f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" Oct 30 21:58:46.123573 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:46.123540 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j\" (UID: \"f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" Oct 30 21:58:46.227642 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:46.227455 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ksg9\" (UniqueName: \"kubernetes.io/projected/f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31-kube-api-access-9ksg9\") pod \"payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j\" (UID: \"f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" Oct 30 21:58:46.227642 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:46.227536 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j\" (UID: \"f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" Oct 30 21:58:46.234728 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:46.228603 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j\" (UID: \"f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" Oct 30 21:58:46.322466 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:46.322418 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ksg9\" (UniqueName: \"kubernetes.io/projected/f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31-kube-api-access-9ksg9\") pod \"payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j\" (UID: \"f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" Oct 30 21:58:46.362240 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:46.362211 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" Oct 30 21:58:46.761113 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:46.761088 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j"] Oct 30 21:58:46.762200 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:58:46.762168 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96f55b5_31e6_4b6a_9dc6_a6ef20a1be31.slice/crio-176449c40837e342582cfae257b96a013a6cede4b728cff8bb4ee870a4f5f352 WatchSource:0}: Error finding container 176449c40837e342582cfae257b96a013a6cede4b728cff8bb4ee870a4f5f352: Status 404 returned error can't find the container with id 176449c40837e342582cfae257b96a013a6cede4b728cff8bb4ee870a4f5f352 Oct 30 21:58:47.028655 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:47.027946 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" event={"ID":"f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31","Type":"ContainerStarted","Data":"cb07f312897257e9ba1b6d348af6b0c97bf28ba42d2441ae8711e0f747173e3e"} Oct 30 21:58:47.028655 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:47.028141 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" event={"ID":"f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31","Type":"ContainerStarted","Data":"176449c40837e342582cfae257b96a013a6cede4b728cff8bb4ee870a4f5f352"} Oct 30 21:58:47.028655 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:47.028196 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" Oct 30 21:58:47.150759 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:47.150711 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" podStartSLOduration=2.150695487 podStartE2EDuration="2.150695487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:47.147487828 +0000 UTC m=+860.247919560" watchObservedRunningTime="2026-10-30 21:58:47.150695487 +0000 UTC m=+860.251127219" Oct 30 21:58:47.921678 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:47.921646 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" Oct 30 21:58:47.922822 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:47.922798 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" Oct 30 21:58:49.616468 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.616139 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl"] Oct 30 21:58:49.620586 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.620568 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl" Oct 30 21:58:49.663774 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.663746 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87dc45da-3324-4fc9-bda4-7186b19d9a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl\" (UID: \"87dc45da-3324-4fc9-bda4-7186b19d9a64\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl" Oct 30 21:58:49.663893 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.663783 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpv74\" (UniqueName: \"kubernetes.io/projected/87dc45da-3324-4fc9-bda4-7186b19d9a64-kube-api-access-bpv74\") pod \"payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl\" (UID: \"87dc45da-3324-4fc9-bda4-7186b19d9a64\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl" Oct 30 21:58:49.674340 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.674315 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-cef074\"" Oct 30 21:58:49.727900 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.727866 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-cef074-dockercfg-6ngz7\"" Oct 30 21:58:49.733144 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.733113 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl"] Oct 30 21:58:49.765369 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.765330 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87dc45da-3324-4fc9-bda4-7186b19d9a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl\" (UID: \"87dc45da-3324-4fc9-bda4-7186b19d9a64\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl" Oct 30 21:58:49.765567 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.765388 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpv74\" (UniqueName: \"kubernetes.io/projected/87dc45da-3324-4fc9-bda4-7186b19d9a64-kube-api-access-bpv74\") pod \"payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl\" (UID: \"87dc45da-3324-4fc9-bda4-7186b19d9a64\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl" Oct 30 21:58:49.766125 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.766098 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87dc45da-3324-4fc9-bda4-7186b19d9a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl\" (UID: \"87dc45da-3324-4fc9-bda4-7186b19d9a64\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl" Oct 30 21:58:49.782273 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.782248 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq"] Oct 30 21:58:49.797460 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.797436 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq" Oct 30 21:58:49.817142 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.817109 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq"] Oct 30 21:58:49.831145 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.831117 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpv74\" (UniqueName: \"kubernetes.io/projected/87dc45da-3324-4fc9-bda4-7186b19d9a64-kube-api-access-bpv74\") pod \"payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl\" (UID: \"87dc45da-3324-4fc9-bda4-7186b19d9a64\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl" Oct 30 21:58:49.866700 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.866606 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ca0831-b8aa-487f-8745-3af2e1d3270e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq\" (UID: \"b3ca0831-b8aa-487f-8745-3af2e1d3270e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq" Oct 30 21:58:49.866825 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.866784 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6769\" (UniqueName: \"kubernetes.io/projected/b3ca0831-b8aa-487f-8745-3af2e1d3270e-kube-api-access-s6769\") pod \"payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq\" (UID: \"b3ca0831-b8aa-487f-8745-3af2e1d3270e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq" Oct 30 21:58:49.935330 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.935295 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl" Oct 30 21:58:49.967721 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.967689 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ca0831-b8aa-487f-8745-3af2e1d3270e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq\" (UID: \"b3ca0831-b8aa-487f-8745-3af2e1d3270e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq" Oct 30 21:58:49.967924 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.967807 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6769\" (UniqueName: \"kubernetes.io/projected/b3ca0831-b8aa-487f-8745-3af2e1d3270e-kube-api-access-s6769\") pod \"payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq\" (UID: \"b3ca0831-b8aa-487f-8745-3af2e1d3270e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq" Oct 30 21:58:49.968511 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:49.968481 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ca0831-b8aa-487f-8745-3af2e1d3270e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq\" (UID: \"b3ca0831-b8aa-487f-8745-3af2e1d3270e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq" Oct 30 21:58:50.044453 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:50.044362 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6769\" (UniqueName: \"kubernetes.io/projected/b3ca0831-b8aa-487f-8745-3af2e1d3270e-kube-api-access-s6769\") pod \"payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq\" (UID: \"b3ca0831-b8aa-487f-8745-3af2e1d3270e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq" Oct 30 21:58:50.120484 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:50.120403 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq" Oct 30 21:58:50.320841 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:50.320812 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl"] Oct 30 21:58:50.322190 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:58:50.322163 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87dc45da_3324_4fc9_bda4_7186b19d9a64.slice/crio-7a3d9f704b415be70322d42712a0a026b3d73c94654897bb1c6b12b184e726f6 WatchSource:0}: Error finding container 7a3d9f704b415be70322d42712a0a026b3d73c94654897bb1c6b12b184e726f6: Status 404 returned error can't find the container with id 7a3d9f704b415be70322d42712a0a026b3d73c94654897bb1c6b12b184e726f6 Oct 30 21:58:50.515879 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:50.515850 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq"] Oct 30 21:58:50.516573 ip-10-0-131-193 kubenswrapper[4335]: W1030 21:58:50.516544 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ca0831_b8aa_487f_8745_3af2e1d3270e.slice/crio-1350007bc532daa33ba8e3aeca6095ff4bac707385c5aeb77bb0e39747a349cc WatchSource:0}: Error finding container 1350007bc532daa33ba8e3aeca6095ff4bac707385c5aeb77bb0e39747a349cc: Status 404 returned error can't find the container with id 1350007bc532daa33ba8e3aeca6095ff4bac707385c5aeb77bb0e39747a349cc Oct 30 21:58:50.942718 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:50.942680 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" Oct 30 21:58:51.064451 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:51.064158 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq" event={"ID":"b3ca0831-b8aa-487f-8745-3af2e1d3270e","Type":"ContainerStarted","Data":"a53179c91698c9b7b03e59aca76cdb7cb7c389a6e356de8e8238a49d67904788"} Oct 30 21:58:51.064451 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:51.064221 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq" Oct 30 21:58:51.064451 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:51.064244 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq" event={"ID":"b3ca0831-b8aa-487f-8745-3af2e1d3270e","Type":"ContainerStarted","Data":"1350007bc532daa33ba8e3aeca6095ff4bac707385c5aeb77bb0e39747a349cc"} Oct 30 21:58:51.066845 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:51.066817 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl" event={"ID":"87dc45da-3324-4fc9-bda4-7186b19d9a64","Type":"ContainerStarted","Data":"9b26e01f073682aab5fed4f5917a247dd4e0c025622c6beaa2d900b1fbab19ee"} Oct 30 21:58:51.066998 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:51.066910 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl" event={"ID":"87dc45da-3324-4fc9-bda4-7186b19d9a64","Type":"ContainerStarted","Data":"7a3d9f704b415be70322d42712a0a026b3d73c94654897bb1c6b12b184e726f6"} Oct 30 21:58:51.067385 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:51.067364 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl" Oct 30 21:58:51.185550 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:51.185489 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq" podStartSLOduration=2.185469844 podStartE2EDuration="2.185469844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:51.179821244 +0000 UTC m=+864.280252976" watchObservedRunningTime="2026-10-30 21:58:51.185469844 +0000 UTC m=+864.285901580" Oct 30 21:58:51.297072 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:51.296984 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl" podStartSLOduration=2.2969704269999998 podStartE2EDuration="2.296970427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:51.294923758 +0000 UTC m=+864.395355505" watchObservedRunningTime="2026-10-30 21:58:51.296970427 +0000 UTC m=+864.397402158" Oct 30 21:58:51.950735 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:51.950699 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" Oct 30 21:58:52.974882 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:52.974794 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" Oct 30 21:58:53.082792 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.082758 4335 generic.go:356] "Generic (PLEG): container finished" podID="aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" containerID="03850171021aacd90821ddd621b0e1ea020bcd503e73f6ffd955d922de2635b8" exitCode=0 Oct 30 21:58:53.082970 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.082837 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" event={"ID":"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1","Type":"ContainerDied","Data":"03850171021aacd90821ddd621b0e1ea020bcd503e73f6ffd955d922de2635b8"} Oct 30 21:58:53.083241 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.083226 4335 scope.go:122] "RemoveContainer" containerID="03850171021aacd90821ddd621b0e1ea020bcd503e73f6ffd955d922de2635b8" Oct 30 21:58:53.084399 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.084372 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:58:53.084483 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.084431 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:58:53.084483 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.084464 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:58:53.174025 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.173983 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b"] Oct 30 21:58:53.178369 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.178345 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" Oct 30 21:58:53.335211 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.334313 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd65776e-f29f-436a-aabb-e9e6b6d3497a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b\" (UID: \"dd65776e-f29f-436a-aabb-e9e6b6d3497a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" Oct 30 21:58:53.335211 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.334460 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdqs9\" (UniqueName: \"kubernetes.io/projected/dd65776e-f29f-436a-aabb-e9e6b6d3497a-kube-api-access-bdqs9\") pod \"payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b\" (UID: \"dd65776e-f29f-436a-aabb-e9e6b6d3497a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" Oct 30 21:58:53.353904 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.353092 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-bf7cbf-dockercfg-v4xp8\"" Oct 30 21:58:53.353904 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.353185 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-bf7cbf\"" Oct 30 21:58:53.361655 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.357154 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b"] Oct 30 21:58:53.435241 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.435210 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdqs9\" (UniqueName: \"kubernetes.io/projected/dd65776e-f29f-436a-aabb-e9e6b6d3497a-kube-api-access-bdqs9\") pod \"payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b\" (UID: \"dd65776e-f29f-436a-aabb-e9e6b6d3497a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" Oct 30 21:58:53.435503 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.435298 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd65776e-f29f-436a-aabb-e9e6b6d3497a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b\" (UID: \"dd65776e-f29f-436a-aabb-e9e6b6d3497a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" Oct 30 21:58:53.436380 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.435972 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd65776e-f29f-436a-aabb-e9e6b6d3497a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b\" (UID: \"dd65776e-f29f-436a-aabb-e9e6b6d3497a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" Oct 30 21:58:53.523510 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.523474 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdqs9\" (UniqueName: \"kubernetes.io/projected/dd65776e-f29f-436a-aabb-e9e6b6d3497a-kube-api-access-bdqs9\") pod \"payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b\" (UID: \"dd65776e-f29f-436a-aabb-e9e6b6d3497a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" Oct 30 21:58:53.524943 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.524708 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" Oct 30 21:58:53.961657 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:53.924681 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b"] Oct 30 21:58:54.091600 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:54.091565 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" event={"ID":"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1","Type":"ContainerStarted","Data":"79029067a8808fea2a9ce5d661bedd9e7430342afe364406d2794e135bf8ab9e"} Oct 30 21:58:54.092955 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:54.092932 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" event={"ID":"dd65776e-f29f-436a-aabb-e9e6b6d3497a","Type":"ContainerStarted","Data":"74d7c26e3b0026844dea7aaa801428e1ef06d17e516740c37a78a329dea345f1"} Oct 30 21:58:54.558244 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:54.555957 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:54.559834 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:54.559801 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:58:54.559966 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:54.559862 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" podUID="aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 21:58:55.116640 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:55.103951 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" event={"ID":"dd65776e-f29f-436a-aabb-e9e6b6d3497a","Type":"ContainerStarted","Data":"3c8b283c688e0a53da0918edb87abe4a24c6323b112bc52e49f7e8a7496267b0"} Oct 30 21:58:55.116640 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:55.104030 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" Oct 30 21:58:55.368977 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:58:55.368895 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:58:55.555101 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:55.555065 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:58:55.555271 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:55.555113 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" podUID="aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 21:58:56.019650 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:56.017208 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" Oct 30 21:58:56.033646 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:56.030759 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" Oct 30 21:58:56.151642 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:56.151560 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" podStartSLOduration=3.151546801 podStartE2EDuration="3.151546801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:55.234783378 +0000 UTC m=+868.335215112" watchObservedRunningTime="2026-10-30 21:58:56.151546801 +0000 UTC m=+869.251978586" Oct 30 21:58:56.554790 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:56.554761 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:58:56.554989 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:56.554814 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" podUID="aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 21:58:57.555461 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:57.555416 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:58:57.556096 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:57.555482 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" podUID="aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 21:58:58.058370 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:58.058339 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" Oct 30 21:58:58.555043 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:58.555012 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:58:58.555346 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:58.555303 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" podUID="aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 21:58:58.842840 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:58.842749 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:59.559339 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:59.559312 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:58:59.560469 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:58:59.560448 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 21:59:02.079730 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:02.079704 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq" Oct 30 21:59:02.205280 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:02.205249 4335 generic.go:356] "Generic (PLEG): container finished" podID="4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" containerID="06b9a0c7bd9b0499ff12b761840ec9569aab185e82eabfa8d9e23ab9c1feb1f9" exitCode=0 Oct 30 21:59:02.205390 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:02.205304 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" event={"ID":"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4","Type":"ContainerDied","Data":"06b9a0c7bd9b0499ff12b761840ec9569aab185e82eabfa8d9e23ab9c1feb1f9"} Oct 30 21:59:02.205667 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:02.205655 4335 scope.go:122] "RemoveContainer" containerID="06b9a0c7bd9b0499ff12b761840ec9569aab185e82eabfa8d9e23ab9c1feb1f9" Oct 30 21:59:02.206910 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:02.206884 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:59:02.206973 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:02.206957 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:59:02.207008 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:02.206994 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:59:03.086048 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:03.086020 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl" Oct 30 21:59:03.273828 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:03.273795 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" event={"ID":"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4","Type":"ContainerStarted","Data":"319f24e86fa6d89dc354a0780de4cb3f4d04e99f39fcb66c3fed9f0f516662be"} Oct 30 21:59:03.341979 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:03.341888 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 21:59:04.235056 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:04.235025 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:59:04.235433 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:04.235213 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:59:04.235433 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:04.235278 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" podUID="4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 21:59:05.235199 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:05.235164 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:59:05.235574 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:05.235219 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" podUID="4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 21:59:06.235141 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:06.235109 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:59:06.235317 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:06.235165 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" podUID="4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 21:59:07.155093 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:07.155067 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" Oct 30 21:59:07.235208 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:07.235182 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:59:07.235368 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:07.235239 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" podUID="4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 21:59:08.234768 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:08.234731 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:59:08.234950 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:08.234796 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" podUID="4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 21:59:09.238928 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:09.238900 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:59:09.239387 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:09.239358 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:59:09.240165 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:09.240146 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 21:59:14.333299 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:14.333260 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 21:59:25.154704 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:25.154670 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 21:59:25.320971 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:59:25.320932 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 21:59:27.160262 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:27.160240 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 21:59:27.160722 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:27.160688 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 21:59:42.063947 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:42.063911 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 21:59:44.594728 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:44.594702 4335 generic.go:356] "Generic (PLEG): container finished" podID="0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" containerID="1b75afff3530fae1c95aa5cd6755bd7ddbdf89a857a28a58ae17a03cb2984e85" exitCode=0 Oct 30 21:59:44.595131 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:44.594781 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" event={"ID":"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec","Type":"ContainerDied","Data":"1b75afff3530fae1c95aa5cd6755bd7ddbdf89a857a28a58ae17a03cb2984e85"} Oct 30 21:59:44.595131 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:44.595093 4335 scope.go:122] "RemoveContainer" containerID="1b75afff3530fae1c95aa5cd6755bd7ddbdf89a857a28a58ae17a03cb2984e85" Oct 30 21:59:44.596410 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:44.596383 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:59:44.596518 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:44.596465 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:59:44.596581 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:44.596515 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:59:45.612651 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:45.612010 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" event={"ID":"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec","Type":"ContainerStarted","Data":"85e87dce410c5854a136b52a2b53bef92c60b8345bd575d338a28aa60dd02927"} Oct 30 21:59:46.221042 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:46.221000 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:59:46.221227 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:46.221118 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:59:46.221227 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:46.221171 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" podUID="0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 21:59:47.220969 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:47.220927 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:59:47.221472 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:47.220984 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" podUID="0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 21:59:48.221026 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:48.220990 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:59:48.221637 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:48.221584 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" podUID="0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 21:59:49.221015 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:49.220981 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:59:49.221199 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:49.221043 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" podUID="0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 21:59:50.220953 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:50.220915 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:59:50.221208 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:50.220983 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" podUID="0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 21:59:51.224110 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:51.224084 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:59:51.224560 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:51.224546 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:59:51.225125 ip-10-0-131-193 kubenswrapper[4335]: I1030 21:59:51.225107 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 21:59:55.320012 ip-10-0-131-193 kubenswrapper[4335]: E1030 21:59:55.319977 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:00:00.184064 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.184033 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj"] Oct 30 22:00:00.189098 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.189071 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" Oct 30 22:00:00.193945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.193921 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9"] Oct 30 22:00:00.200597 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.200577 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" Oct 30 22:00:00.201849 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.201821 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z"] Oct 30 22:00:00.209843 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.209811 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" Oct 30 22:00:00.233536 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.233513 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv"] Oct 30 22:00:00.237058 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.237041 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj"] Oct 30 22:00:00.237184 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.237163 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" Oct 30 22:00:00.240582 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.240566 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" Oct 30 22:00:00.241285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.241119 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:00:00.241285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.241127 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:00:00.241285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.241189 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:00:00.289027 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.288659 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj"] Oct 30 22:00:00.291988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.291965 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9"] Oct 30 22:00:00.294853 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.294834 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z"] Oct 30 22:00:00.294937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.294897 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5809eb74-cd66-41b0-b527-fb8fcbc722ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9\" (UID: \"5809eb74-cd66-41b0-b527-fb8fcbc722ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" Oct 30 22:00:00.294937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.294927 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5trs\" (UniqueName: \"kubernetes.io/projected/40092c87-92a3-48f7-8a2c-64f75215df63-kube-api-access-t5trs\") pod \"maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z\" (UID: \"40092c87-92a3-48f7-8a2c-64f75215df63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" Oct 30 22:00:00.295029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.294960 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpt6g\" (UniqueName: \"kubernetes.io/projected/5809eb74-cd66-41b0-b527-fb8fcbc722ae-kube-api-access-tpt6g\") pod \"maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9\" (UID: \"5809eb74-cd66-41b0-b527-fb8fcbc722ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" Oct 30 22:00:00.295067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.295043 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4221a80-5591-41ae-86e3-ff85b736b65e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-89nsj\" (UID: \"d4221a80-5591-41ae-86e3-ff85b736b65e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" Oct 30 22:00:00.295124 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.295106 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40092c87-92a3-48f7-8a2c-64f75215df63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z\" (UID: \"40092c87-92a3-48f7-8a2c-64f75215df63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" Oct 30 22:00:00.295165 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.295150 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlvjm\" (UniqueName: \"kubernetes.io/projected/d4221a80-5591-41ae-86e3-ff85b736b65e-kube-api-access-qlvjm\") pod \"maas-api-key-cleanup-29889960-89nsj\" (UID: \"d4221a80-5591-41ae-86e3-ff85b736b65e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" Oct 30 22:00:00.295214 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.295200 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7753b052-e934-45b5-a8ca-b73f3fe0dddc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv\" (UID: \"7753b052-e934-45b5-a8ca-b73f3fe0dddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" Oct 30 22:00:00.295275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.295263 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j5wc\" (UniqueName: \"kubernetes.io/projected/79510ae6-8c22-45e7-98d2-d6174b001098-kube-api-access-4j5wc\") pod \"maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj\" (UID: \"79510ae6-8c22-45e7-98d2-d6174b001098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" Oct 30 22:00:00.295309 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.295298 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79510ae6-8c22-45e7-98d2-d6174b001098-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj\" (UID: \"79510ae6-8c22-45e7-98d2-d6174b001098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" Oct 30 22:00:00.295449 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.295428 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7pcw\" (UniqueName: \"kubernetes.io/projected/7753b052-e934-45b5-a8ca-b73f3fe0dddc-kube-api-access-n7pcw\") pod \"maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv\" (UID: \"7753b052-e934-45b5-a8ca-b73f3fe0dddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" Oct 30 22:00:00.334274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.334248 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv"] Oct 30 22:00:00.341636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.341285 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj"] Oct 30 22:00:00.396117 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.396091 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5809eb74-cd66-41b0-b527-fb8fcbc722ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9\" (UID: \"5809eb74-cd66-41b0-b527-fb8fcbc722ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" Oct 30 22:00:00.396242 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.396133 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5trs\" (UniqueName: \"kubernetes.io/projected/40092c87-92a3-48f7-8a2c-64f75215df63-kube-api-access-t5trs\") pod \"maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z\" (UID: \"40092c87-92a3-48f7-8a2c-64f75215df63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" Oct 30 22:00:00.396242 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.396165 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpt6g\" (UniqueName: \"kubernetes.io/projected/5809eb74-cd66-41b0-b527-fb8fcbc722ae-kube-api-access-tpt6g\") pod \"maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9\" (UID: \"5809eb74-cd66-41b0-b527-fb8fcbc722ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" Oct 30 22:00:00.396242 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.396217 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4221a80-5591-41ae-86e3-ff85b736b65e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-89nsj\" (UID: \"d4221a80-5591-41ae-86e3-ff85b736b65e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" Oct 30 22:00:00.396379 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.396286 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40092c87-92a3-48f7-8a2c-64f75215df63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z\" (UID: \"40092c87-92a3-48f7-8a2c-64f75215df63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" Oct 30 22:00:00.396379 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.396328 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlvjm\" (UniqueName: \"kubernetes.io/projected/d4221a80-5591-41ae-86e3-ff85b736b65e-kube-api-access-qlvjm\") pod \"maas-api-key-cleanup-29889960-89nsj\" (UID: \"d4221a80-5591-41ae-86e3-ff85b736b65e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" Oct 30 22:00:00.396482 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.396390 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7753b052-e934-45b5-a8ca-b73f3fe0dddc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv\" (UID: \"7753b052-e934-45b5-a8ca-b73f3fe0dddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" Oct 30 22:00:00.396482 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.396457 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j5wc\" (UniqueName: \"kubernetes.io/projected/79510ae6-8c22-45e7-98d2-d6174b001098-kube-api-access-4j5wc\") pod \"maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj\" (UID: \"79510ae6-8c22-45e7-98d2-d6174b001098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" Oct 30 22:00:00.397008 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.396986 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5809eb74-cd66-41b0-b527-fb8fcbc722ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9\" (UID: \"5809eb74-cd66-41b0-b527-fb8fcbc722ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" Oct 30 22:00:00.397186 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.397163 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40092c87-92a3-48f7-8a2c-64f75215df63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z\" (UID: \"40092c87-92a3-48f7-8a2c-64f75215df63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" Oct 30 22:00:00.397186 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.397175 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4221a80-5591-41ae-86e3-ff85b736b65e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-89nsj\" (UID: \"d4221a80-5591-41ae-86e3-ff85b736b65e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" Oct 30 22:00:00.397405 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.397389 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7753b052-e934-45b5-a8ca-b73f3fe0dddc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv\" (UID: \"7753b052-e934-45b5-a8ca-b73f3fe0dddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" Oct 30 22:00:00.397976 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.397953 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79510ae6-8c22-45e7-98d2-d6174b001098-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj\" (UID: \"79510ae6-8c22-45e7-98d2-d6174b001098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" Oct 30 22:00:00.398090 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.398071 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7pcw\" (UniqueName: \"kubernetes.io/projected/7753b052-e934-45b5-a8ca-b73f3fe0dddc-kube-api-access-n7pcw\") pod \"maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv\" (UID: \"7753b052-e934-45b5-a8ca-b73f3fe0dddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" Oct 30 22:00:00.398588 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.398569 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79510ae6-8c22-45e7-98d2-d6174b001098-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj\" (UID: \"79510ae6-8c22-45e7-98d2-d6174b001098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" Oct 30 22:00:00.457399 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.457331 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpt6g\" (UniqueName: \"kubernetes.io/projected/5809eb74-cd66-41b0-b527-fb8fcbc722ae-kube-api-access-tpt6g\") pod \"maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9\" (UID: \"5809eb74-cd66-41b0-b527-fb8fcbc722ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" Oct 30 22:00:00.457561 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.457538 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j5wc\" (UniqueName: \"kubernetes.io/projected/79510ae6-8c22-45e7-98d2-d6174b001098-kube-api-access-4j5wc\") pod \"maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj\" (UID: \"79510ae6-8c22-45e7-98d2-d6174b001098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" Oct 30 22:00:00.457677 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.457538 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5trs\" (UniqueName: \"kubernetes.io/projected/40092c87-92a3-48f7-8a2c-64f75215df63-kube-api-access-t5trs\") pod \"maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z\" (UID: \"40092c87-92a3-48f7-8a2c-64f75215df63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" Oct 30 22:00:00.457881 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.457857 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlvjm\" (UniqueName: \"kubernetes.io/projected/d4221a80-5591-41ae-86e3-ff85b736b65e-kube-api-access-qlvjm\") pod \"maas-api-key-cleanup-29889960-89nsj\" (UID: \"d4221a80-5591-41ae-86e3-ff85b736b65e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" Oct 30 22:00:00.458258 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.458241 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7pcw\" (UniqueName: \"kubernetes.io/projected/7753b052-e934-45b5-a8ca-b73f3fe0dddc-kube-api-access-n7pcw\") pod \"maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv\" (UID: \"7753b052-e934-45b5-a8ca-b73f3fe0dddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" Oct 30 22:00:00.508522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.508500 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" Oct 30 22:00:00.521847 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.521828 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" Oct 30 22:00:00.528724 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.528691 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" Oct 30 22:00:00.552231 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.552201 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" Oct 30 22:00:00.558663 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.558484 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" Oct 30 22:00:00.907328 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:00:00.907301 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79510ae6_8c22_45e7_98d2_d6174b001098.slice/crio-bed7e4fb6a139a8cf85a36869fa2a3397c8177ce03924f998994b51ac16b1070 WatchSource:0}: Error finding container bed7e4fb6a139a8cf85a36869fa2a3397c8177ce03924f998994b51ac16b1070: Status 404 returned error can't find the container with id bed7e4fb6a139a8cf85a36869fa2a3397c8177ce03924f998994b51ac16b1070 Oct 30 22:00:00.907691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.907668 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj"] Oct 30 22:00:00.957944 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:00:00.957913 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40092c87_92a3_48f7_8a2c_64f75215df63.slice/crio-4abb7d4c5f416ac0365a048c17568e11ec327c14ea4524ebfd87058e2c79c8ad WatchSource:0}: Error finding container 4abb7d4c5f416ac0365a048c17568e11ec327c14ea4524ebfd87058e2c79c8ad: Status 404 returned error can't find the container with id 4abb7d4c5f416ac0365a048c17568e11ec327c14ea4524ebfd87058e2c79c8ad Oct 30 22:00:00.958780 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:00:00.958761 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5809eb74_cd66_41b0_b527_fb8fcbc722ae.slice/crio-e22de2910578c1625e8dc95cec7bf66478bb34f7d80f2787a18aa7da0254772c WatchSource:0}: Error finding container e22de2910578c1625e8dc95cec7bf66478bb34f7d80f2787a18aa7da0254772c: Status 404 returned error can't find the container with id e22de2910578c1625e8dc95cec7bf66478bb34f7d80f2787a18aa7da0254772c Oct 30 22:00:00.959372 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.959351 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9"] Oct 30 22:00:00.963733 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:00.963714 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z"] Oct 30 22:00:01.011061 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:00:01.011027 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7753b052_e934_45b5_a8ca_b73f3fe0dddc.slice/crio-9ca9207eb5cc4a1c2ca0d270185f69b5a46319f6624057abf6e6df2e9c57192d WatchSource:0}: Error finding container 9ca9207eb5cc4a1c2ca0d270185f69b5a46319f6624057abf6e6df2e9c57192d: Status 404 returned error can't find the container with id 9ca9207eb5cc4a1c2ca0d270185f69b5a46319f6624057abf6e6df2e9c57192d Oct 30 22:00:01.011253 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:01.011199 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv"] Oct 30 22:00:01.011859 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:00:01.011835 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4221a80_5591_41ae_86e3_ff85b736b65e.slice/crio-1054c191ff07060480501dadf3626dc5d8c0d3d480b5d64d84c62c21fc96cafd WatchSource:0}: Error finding container 1054c191ff07060480501dadf3626dc5d8c0d3d480b5d64d84c62c21fc96cafd: Status 404 returned error can't find the container with id 1054c191ff07060480501dadf3626dc5d8c0d3d480b5d64d84c62c21fc96cafd Oct 30 22:00:01.015642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:01.014710 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj"] Oct 30 22:00:01.305529 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:01.305490 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:00:01.731213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:01.731130 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" event={"ID":"7753b052-e934-45b5-a8ca-b73f3fe0dddc","Type":"ContainerStarted","Data":"9ca9207eb5cc4a1c2ca0d270185f69b5a46319f6624057abf6e6df2e9c57192d"} Oct 30 22:00:01.732501 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:01.732473 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" event={"ID":"79510ae6-8c22-45e7-98d2-d6174b001098","Type":"ContainerStarted","Data":"bed7e4fb6a139a8cf85a36869fa2a3397c8177ce03924f998994b51ac16b1070"} Oct 30 22:00:01.734291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:01.734254 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" event={"ID":"d4221a80-5591-41ae-86e3-ff85b736b65e","Type":"ContainerStarted","Data":"1054c191ff07060480501dadf3626dc5d8c0d3d480b5d64d84c62c21fc96cafd"} Oct 30 22:00:01.735863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:01.735840 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" event={"ID":"40092c87-92a3-48f7-8a2c-64f75215df63","Type":"ContainerStarted","Data":"4abb7d4c5f416ac0365a048c17568e11ec327c14ea4524ebfd87058e2c79c8ad"} Oct 30 22:00:01.737028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:01.737010 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" event={"ID":"5809eb74-cd66-41b0-b527-fb8fcbc722ae","Type":"ContainerStarted","Data":"e22de2910578c1625e8dc95cec7bf66478bb34f7d80f2787a18aa7da0254772c"} Oct 30 22:00:07.790503 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:07.790470 4335 generic.go:356] "Generic (PLEG): container finished" podID="7753b052-e934-45b5-a8ca-b73f3fe0dddc" containerID="7b9fb8c53d955d7b799beb2c5b927e1514ba975186f8d4520891be102490f009" exitCode=60 Oct 30 22:00:07.790938 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:07.790567 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" event={"ID":"7753b052-e934-45b5-a8ca-b73f3fe0dddc","Type":"ContainerDied","Data":"7b9fb8c53d955d7b799beb2c5b927e1514ba975186f8d4520891be102490f009"} Oct 30 22:00:07.790938 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:07.790829 4335 scope.go:122] "RemoveContainer" containerID="7b9fb8c53d955d7b799beb2c5b927e1514ba975186f8d4520891be102490f009" Oct 30 22:00:07.792676 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:07.792591 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" event={"ID":"79510ae6-8c22-45e7-98d2-d6174b001098","Type":"ContainerStarted","Data":"2a213637a6ba49192ffc6145aec037b24d11667cfdddb47d419e6fdeac29f5ef"} Oct 30 22:00:07.795042 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:07.795015 4335 generic.go:356] "Generic (PLEG): container finished" podID="d4221a80-5591-41ae-86e3-ff85b736b65e" containerID="d753b1ce1ad5b4b897890547dc60ee16f98b07b8f6d53e37fa7faed1752cf5d4" exitCode=60 Oct 30 22:00:07.795173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:07.795145 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" event={"ID":"d4221a80-5591-41ae-86e3-ff85b736b65e","Type":"ContainerDied","Data":"d753b1ce1ad5b4b897890547dc60ee16f98b07b8f6d53e37fa7faed1752cf5d4"} Oct 30 22:00:07.795337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:07.795314 4335 scope.go:122] "RemoveContainer" containerID="d753b1ce1ad5b4b897890547dc60ee16f98b07b8f6d53e37fa7faed1752cf5d4" Oct 30 22:00:07.798073 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:07.797545 4335 generic.go:356] "Generic (PLEG): container finished" podID="40092c87-92a3-48f7-8a2c-64f75215df63" containerID="ba0c3e1686287e7521b4df80d228ecd7de1e85eef105768f9c29e47a8d7ac306" exitCode=60 Oct 30 22:00:07.798073 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:07.797687 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" event={"ID":"40092c87-92a3-48f7-8a2c-64f75215df63","Type":"ContainerDied","Data":"ba0c3e1686287e7521b4df80d228ecd7de1e85eef105768f9c29e47a8d7ac306"} Oct 30 22:00:07.798073 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:07.797842 4335 scope.go:122] "RemoveContainer" containerID="ba0c3e1686287e7521b4df80d228ecd7de1e85eef105768f9c29e47a8d7ac306" Oct 30 22:00:07.799352 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:07.799303 4335 generic.go:356] "Generic (PLEG): container finished" podID="5809eb74-cd66-41b0-b527-fb8fcbc722ae" containerID="5c7d9c59838e10e58563065a7a5a1a755f260c8e37645bbfe417cd5afc7c9feb" exitCode=60 Oct 30 22:00:07.799352 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:07.799348 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" event={"ID":"5809eb74-cd66-41b0-b527-fb8fcbc722ae","Type":"ContainerDied","Data":"5c7d9c59838e10e58563065a7a5a1a755f260c8e37645bbfe417cd5afc7c9feb"} Oct 30 22:00:07.799582 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:07.799563 4335 scope.go:122] "RemoveContainer" containerID="5c7d9c59838e10e58563065a7a5a1a755f260c8e37645bbfe417cd5afc7c9feb" Oct 30 22:00:08.141533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:08.141471 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" podStartSLOduration=2.353971649 podStartE2EDuration="8.141453861s" totalImagesPullingTime="5.787482212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:08.137445495 +0000 UTC m=+941.237877228" watchObservedRunningTime="2026-10-30 22:00:08.141453861 +0000 UTC m=+941.241885594" Oct 30 22:00:08.809910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:08.809874 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" event={"ID":"7753b052-e934-45b5-a8ca-b73f3fe0dddc","Type":"ContainerStarted","Data":"0093266911ec86e9180e5f5d77f8b61c58e6dfee6426e8c391b12499a49d1a7d"} Oct 30 22:00:08.811743 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:08.811712 4335 generic.go:356] "Generic (PLEG): container finished" podID="79510ae6-8c22-45e7-98d2-d6174b001098" containerID="2a213637a6ba49192ffc6145aec037b24d11667cfdddb47d419e6fdeac29f5ef" exitCode=60 Oct 30 22:00:08.811878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:08.811785 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" event={"ID":"79510ae6-8c22-45e7-98d2-d6174b001098","Type":"ContainerDied","Data":"2a213637a6ba49192ffc6145aec037b24d11667cfdddb47d419e6fdeac29f5ef"} Oct 30 22:00:08.811998 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:08.811980 4335 scope.go:122] "RemoveContainer" containerID="2a213637a6ba49192ffc6145aec037b24d11667cfdddb47d419e6fdeac29f5ef" Oct 30 22:00:08.814156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:08.814119 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" event={"ID":"d4221a80-5591-41ae-86e3-ff85b736b65e","Type":"ContainerStarted","Data":"8530facd436971e2835c07f0c239ee11665af4c1fedad7ae9a33cc9830debe8a"} Oct 30 22:00:08.816744 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:08.816662 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" event={"ID":"40092c87-92a3-48f7-8a2c-64f75215df63","Type":"ContainerStarted","Data":"2e45f7298454af8a135c26de9abc1ad6cccab58cd3902467e7694c0c9dfee06f"} Oct 30 22:00:08.818895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:08.818871 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" event={"ID":"5809eb74-cd66-41b0-b527-fb8fcbc722ae","Type":"ContainerStarted","Data":"efef621e9a0fc44a94f05db9c3246bac747f47e5f1422678dfbbe70db783b582"} Oct 30 22:00:08.926317 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:08.926271 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" podStartSLOduration=3.392078337 podStartE2EDuration="8.926258589s" totalImagesPullingTime="5.534180252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:08.925359263 +0000 UTC m=+942.025790994" watchObservedRunningTime="2026-10-30 22:00:08.926258589 +0000 UTC m=+942.026690321" Oct 30 22:00:09.039208 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.039160 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" podStartSLOduration=3.325695852 podStartE2EDuration="9.039146611s" totalImagesPullingTime="5.713450759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:09.03891793 +0000 UTC m=+942.139349664" watchObservedRunningTime="2026-10-30 22:00:09.039146611 +0000 UTC m=+942.139578343" Oct 30 22:00:09.154864 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.154770 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" podStartSLOduration=3.442069709 podStartE2EDuration="9.154750869s" totalImagesPullingTime="5.71268116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:09.152754096 +0000 UTC m=+942.253185829" watchObservedRunningTime="2026-10-30 22:00:09.154750869 +0000 UTC m=+942.255182602" Oct 30 22:00:09.267779 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.267734 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" podStartSLOduration=3.637316581 podStartE2EDuration="9.267720161s" totalImagesPullingTime="5.63040358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:09.266504076 +0000 UTC m=+942.366935818" watchObservedRunningTime="2026-10-30 22:00:09.267720161 +0000 UTC m=+942.368151894" Oct 30 22:00:09.826864 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.826836 4335 generic.go:356] "Generic (PLEG): container finished" podID="7753b052-e934-45b5-a8ca-b73f3fe0dddc" containerID="0093266911ec86e9180e5f5d77f8b61c58e6dfee6426e8c391b12499a49d1a7d" exitCode=60 Oct 30 22:00:09.827222 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.826874 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" event={"ID":"7753b052-e934-45b5-a8ca-b73f3fe0dddc","Type":"ContainerDied","Data":"0093266911ec86e9180e5f5d77f8b61c58e6dfee6426e8c391b12499a49d1a7d"} Oct 30 22:00:09.827222 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.826973 4335 scope.go:122] "RemoveContainer" containerID="7b9fb8c53d955d7b799beb2c5b927e1514ba975186f8d4520891be102490f009" Oct 30 22:00:09.827222 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.827200 4335 scope.go:122] "RemoveContainer" containerID="0093266911ec86e9180e5f5d77f8b61c58e6dfee6426e8c391b12499a49d1a7d" Oct 30 22:00:09.827494 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:00:09.827472 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv_odh-ai-gateway-infra(7753b052-e934-45b5-a8ca-b73f3fe0dddc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" podUID="7753b052-e934-45b5-a8ca-b73f3fe0dddc" Oct 30 22:00:09.829059 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.829039 4335 generic.go:356] "Generic (PLEG): container finished" podID="79510ae6-8c22-45e7-98d2-d6174b001098" containerID="4146d2b15ef2cc5f0db48b656e699723f266408d4c67684b758f6003d024bd15" exitCode=60 Oct 30 22:00:09.829156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.829122 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" event={"ID":"79510ae6-8c22-45e7-98d2-d6174b001098","Type":"ContainerDied","Data":"4146d2b15ef2cc5f0db48b656e699723f266408d4c67684b758f6003d024bd15"} Oct 30 22:00:09.829344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.829328 4335 scope.go:122] "RemoveContainer" containerID="4146d2b15ef2cc5f0db48b656e699723f266408d4c67684b758f6003d024bd15" Oct 30 22:00:09.829580 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:00:09.829560 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj_odh-ai-gateway-infra(79510ae6-8c22-45e7-98d2-d6174b001098)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" podUID="79510ae6-8c22-45e7-98d2-d6174b001098" Oct 30 22:00:09.831051 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.831029 4335 generic.go:356] "Generic (PLEG): container finished" podID="d4221a80-5591-41ae-86e3-ff85b736b65e" containerID="8530facd436971e2835c07f0c239ee11665af4c1fedad7ae9a33cc9830debe8a" exitCode=60 Oct 30 22:00:09.831129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.831110 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" event={"ID":"d4221a80-5591-41ae-86e3-ff85b736b65e","Type":"ContainerDied","Data":"8530facd436971e2835c07f0c239ee11665af4c1fedad7ae9a33cc9830debe8a"} Oct 30 22:00:09.831310 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.831293 4335 scope.go:122] "RemoveContainer" containerID="8530facd436971e2835c07f0c239ee11665af4c1fedad7ae9a33cc9830debe8a" Oct 30 22:00:09.831594 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:00:09.831575 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-89nsj_odh-ai-gateway-infra(d4221a80-5591-41ae-86e3-ff85b736b65e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" podUID="d4221a80-5591-41ae-86e3-ff85b736b65e" Oct 30 22:00:09.833191 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.833167 4335 generic.go:356] "Generic (PLEG): container finished" podID="40092c87-92a3-48f7-8a2c-64f75215df63" containerID="2e45f7298454af8a135c26de9abc1ad6cccab58cd3902467e7694c0c9dfee06f" exitCode=60 Oct 30 22:00:09.833431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.833408 4335 scope.go:122] "RemoveContainer" containerID="2e45f7298454af8a135c26de9abc1ad6cccab58cd3902467e7694c0c9dfee06f" Oct 30 22:00:09.833532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.833509 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" event={"ID":"40092c87-92a3-48f7-8a2c-64f75215df63","Type":"ContainerDied","Data":"2e45f7298454af8a135c26de9abc1ad6cccab58cd3902467e7694c0c9dfee06f"} Oct 30 22:00:09.833662 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:00:09.833641 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z_odh-ai-gateway-infra(40092c87-92a3-48f7-8a2c-64f75215df63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" podUID="40092c87-92a3-48f7-8a2c-64f75215df63" Oct 30 22:00:09.835301 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.835282 4335 generic.go:356] "Generic (PLEG): container finished" podID="5809eb74-cd66-41b0-b527-fb8fcbc722ae" containerID="efef621e9a0fc44a94f05db9c3246bac747f47e5f1422678dfbbe70db783b582" exitCode=60 Oct 30 22:00:09.835397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.835318 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" event={"ID":"5809eb74-cd66-41b0-b527-fb8fcbc722ae","Type":"ContainerDied","Data":"efef621e9a0fc44a94f05db9c3246bac747f47e5f1422678dfbbe70db783b582"} Oct 30 22:00:09.835499 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.835484 4335 scope.go:122] "RemoveContainer" containerID="efef621e9a0fc44a94f05db9c3246bac747f47e5f1422678dfbbe70db783b582" Oct 30 22:00:09.835719 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:00:09.835701 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9_odh-ai-gateway-infra(5809eb74-cd66-41b0-b527-fb8fcbc722ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" podUID="5809eb74-cd66-41b0-b527-fb8fcbc722ae" Oct 30 22:00:09.857894 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.857877 4335 scope.go:122] "RemoveContainer" containerID="2a213637a6ba49192ffc6145aec037b24d11667cfdddb47d419e6fdeac29f5ef" Oct 30 22:00:09.869993 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.869974 4335 scope.go:122] "RemoveContainer" containerID="d753b1ce1ad5b4b897890547dc60ee16f98b07b8f6d53e37fa7faed1752cf5d4" Oct 30 22:00:09.883300 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.883281 4335 scope.go:122] "RemoveContainer" containerID="ba0c3e1686287e7521b4df80d228ecd7de1e85eef105768f9c29e47a8d7ac306" Oct 30 22:00:09.895692 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:09.895673 4335 scope.go:122] "RemoveContainer" containerID="5c7d9c59838e10e58563065a7a5a1a755f260c8e37645bbfe417cd5afc7c9feb" Oct 30 22:00:10.844007 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:10.843978 4335 scope.go:122] "RemoveContainer" containerID="0093266911ec86e9180e5f5d77f8b61c58e6dfee6426e8c391b12499a49d1a7d" Oct 30 22:00:10.844418 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:00:10.844193 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv_odh-ai-gateway-infra(7753b052-e934-45b5-a8ca-b73f3fe0dddc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" podUID="7753b052-e934-45b5-a8ca-b73f3fe0dddc" Oct 30 22:00:10.845692 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:10.845673 4335 scope.go:122] "RemoveContainer" containerID="4146d2b15ef2cc5f0db48b656e699723f266408d4c67684b758f6003d024bd15" Oct 30 22:00:10.845882 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:00:10.845862 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj_odh-ai-gateway-infra(79510ae6-8c22-45e7-98d2-d6174b001098)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" podUID="79510ae6-8c22-45e7-98d2-d6174b001098" Oct 30 22:00:10.847210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:10.847193 4335 scope.go:122] "RemoveContainer" containerID="8530facd436971e2835c07f0c239ee11665af4c1fedad7ae9a33cc9830debe8a" Oct 30 22:00:10.847403 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:00:10.847386 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-89nsj_odh-ai-gateway-infra(d4221a80-5591-41ae-86e3-ff85b736b65e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" podUID="d4221a80-5591-41ae-86e3-ff85b736b65e" Oct 30 22:00:10.848653 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:10.848639 4335 scope.go:122] "RemoveContainer" containerID="2e45f7298454af8a135c26de9abc1ad6cccab58cd3902467e7694c0c9dfee06f" Oct 30 22:00:10.848842 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:00:10.848822 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z_odh-ai-gateway-infra(40092c87-92a3-48f7-8a2c-64f75215df63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" podUID="40092c87-92a3-48f7-8a2c-64f75215df63" Oct 30 22:00:10.850164 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:10.850150 4335 scope.go:122] "RemoveContainer" containerID="efef621e9a0fc44a94f05db9c3246bac747f47e5f1422678dfbbe70db783b582" Oct 30 22:00:10.850320 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:00:10.850304 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9_odh-ai-gateway-infra(5809eb74-cd66-41b0-b527-fb8fcbc722ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" podUID="5809eb74-cd66-41b0-b527-fb8fcbc722ae" Oct 30 22:00:16.532506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.532471 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7"] Oct 30 22:00:16.536783 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.536760 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.587717 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.587693 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgrtd\" (UniqueName: \"kubernetes.io/projected/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-kube-api-access-fgrtd\") pod \"simulator-w4-536975-kserve-785485744-p27c7\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.587840 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.587736 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-tmp-dir\") pod \"simulator-w4-536975-kserve-785485744-p27c7\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.587840 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.587756 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-tls-certs\") pod \"simulator-w4-536975-kserve-785485744-p27c7\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.587912 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.587870 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-model-cache\") pod \"simulator-w4-536975-kserve-785485744-p27c7\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.587946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.587931 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-home\") pod \"simulator-w4-536975-kserve-785485744-p27c7\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.587982 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.587952 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-dshm\") pod \"simulator-w4-536975-kserve-785485744-p27c7\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.639862 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.639823 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-536975\"/\"openshift-service-ca.crt\"" Oct 30 22:00:16.640011 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.639903 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-536975\"/\"kube-root-ca.crt\"" Oct 30 22:00:16.640076 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.640050 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-536975\"/\"simulator-w4-536975-kserve-self-signed-certs\"" Oct 30 22:00:16.640076 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.640057 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-536975\"/\"default-dockercfg-mxbdq\"" Oct 30 22:00:16.643391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.643372 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7"] Oct 30 22:00:16.691802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.689420 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrtd\" (UniqueName: \"kubernetes.io/projected/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-kube-api-access-fgrtd\") pod \"simulator-w4-536975-kserve-785485744-p27c7\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.691802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.689478 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-tmp-dir\") pod \"simulator-w4-536975-kserve-785485744-p27c7\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.691802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.689503 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-tls-certs\") pod \"simulator-w4-536975-kserve-785485744-p27c7\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.691802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.689546 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-model-cache\") pod \"simulator-w4-536975-kserve-785485744-p27c7\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.691802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.689586 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-home\") pod \"simulator-w4-536975-kserve-785485744-p27c7\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.691802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.689631 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-dshm\") pod \"simulator-w4-536975-kserve-785485744-p27c7\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.692921 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.692876 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-model-cache\") pod \"simulator-w4-536975-kserve-785485744-p27c7\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.693449 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.693432 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-home\") pod \"simulator-w4-536975-kserve-785485744-p27c7\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.697749 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.694316 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-dshm\") pod \"simulator-w4-536975-kserve-785485744-p27c7\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.697749 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.694999 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-tmp-dir\") pod \"simulator-w4-536975-kserve-785485744-p27c7\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.701120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.701098 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-tls-certs\") pod \"simulator-w4-536975-kserve-785485744-p27c7\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.754798 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.754772 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgrtd\" (UniqueName: \"kubernetes.io/projected/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-kube-api-access-fgrtd\") pod \"simulator-w4-536975-kserve-785485744-p27c7\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.848943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.848917 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:16.861926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:16.861905 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:00:17.228411 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:17.228376 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7"] Oct 30 22:00:17.228881 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:00:17.228856 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a0129d4_b54f_4081_9f28_3bc5c61dbf17.slice/crio-2f089028b156b635c4a9871307b9acc4ea00efe07fc0027e406b2a12f2127d84 WatchSource:0}: Error finding container 2f089028b156b635c4a9871307b9acc4ea00efe07fc0027e406b2a12f2127d84: Status 404 returned error can't find the container with id 2f089028b156b635c4a9871307b9acc4ea00efe07fc0027e406b2a12f2127d84 Oct 30 22:00:17.898422 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:17.898389 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" event={"ID":"9a0129d4-b54f-4081-9f28-3bc5c61dbf17","Type":"ContainerStarted","Data":"1ed59dcbc635e21756cadafbf5a84a3796ac53e87950eb92b137b3f7cda3682f"} Oct 30 22:00:17.898824 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:17.898434 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" event={"ID":"9a0129d4-b54f-4081-9f28-3bc5c61dbf17","Type":"ContainerStarted","Data":"2f089028b156b635c4a9871307b9acc4ea00efe07fc0027e406b2a12f2127d84"} Oct 30 22:00:18.016063 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:18.016015 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podStartSLOduration=2.016001203 podStartE2EDuration="2.016001203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:18.013610424 +0000 UTC m=+951.114042157" watchObservedRunningTime="2026-10-30 22:00:18.016001203 +0000 UTC m=+951.116432952" Oct 30 22:00:18.849603 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:18.849573 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:19.057639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:19.057588 4335 scope.go:122] "RemoveContainer" containerID="2e45f7298454af8a135c26de9abc1ad6cccab58cd3902467e7694c0c9dfee06f" Oct 30 22:00:19.915034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:19.915007 4335 generic.go:356] "Generic (PLEG): container finished" podID="40092c87-92a3-48f7-8a2c-64f75215df63" containerID="08ef1ebc0e69c75c6e37ae737c23d624922be3384233d276e1358c218d9da675" exitCode=60 Oct 30 22:00:19.915144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:19.915077 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" event={"ID":"40092c87-92a3-48f7-8a2c-64f75215df63","Type":"ContainerDied","Data":"08ef1ebc0e69c75c6e37ae737c23d624922be3384233d276e1358c218d9da675"} Oct 30 22:00:19.915144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:19.915136 4335 scope.go:122] "RemoveContainer" containerID="2e45f7298454af8a135c26de9abc1ad6cccab58cd3902467e7694c0c9dfee06f" Oct 30 22:00:19.915607 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:19.915579 4335 scope.go:122] "RemoveContainer" containerID="08ef1ebc0e69c75c6e37ae737c23d624922be3384233d276e1358c218d9da675" Oct 30 22:00:19.915891 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:00:19.915849 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z_odh-ai-gateway-infra(40092c87-92a3-48f7-8a2c-64f75215df63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" podUID="40092c87-92a3-48f7-8a2c-64f75215df63" Oct 30 22:00:20.050697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:20.050672 4335 scope.go:122] "RemoveContainer" containerID="efef621e9a0fc44a94f05db9c3246bac747f47e5f1422678dfbbe70db783b582" Oct 30 22:00:20.933700 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:20.933669 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" event={"ID":"5809eb74-cd66-41b0-b527-fb8fcbc722ae","Type":"ContainerStarted","Data":"66c5dbf08cbb2691aa87a4923fc1470dd55e0b91c7df5fd81d966a706522ad44"} Oct 30 22:00:21.046772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.046746 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z"] Oct 30 22:00:21.050168 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.050151 4335 scope.go:122] "RemoveContainer" containerID="4146d2b15ef2cc5f0db48b656e699723f266408d4c67684b758f6003d024bd15" Oct 30 22:00:21.197778 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.197722 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" Oct 30 22:00:21.343660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.340008 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40092c87-92a3-48f7-8a2c-64f75215df63-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40092c87-92a3-48f7-8a2c-64f75215df63-maas-api-service-ca\") pod \"40092c87-92a3-48f7-8a2c-64f75215df63\" (UID: \"40092c87-92a3-48f7-8a2c-64f75215df63\") " Oct 30 22:00:21.343660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.340203 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40092c87-92a3-48f7-8a2c-64f75215df63-kube-api-access-t5trs\" (UniqueName: \"kubernetes.io/projected/40092c87-92a3-48f7-8a2c-64f75215df63-kube-api-access-t5trs\") pod \"40092c87-92a3-48f7-8a2c-64f75215df63\" (UID: \"40092c87-92a3-48f7-8a2c-64f75215df63\") " Oct 30 22:00:21.343660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.340383 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40092c87-92a3-48f7-8a2c-64f75215df63-maas-api-service-ca" pod "40092c87-92a3-48f7-8a2c-64f75215df63" (UID: "40092c87-92a3-48f7-8a2c-64f75215df63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:21.343660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.340547 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40092c87-92a3-48f7-8a2c-64f75215df63-maas-api-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.343660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.342526 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40092c87-92a3-48f7-8a2c-64f75215df63-kube-api-access-t5trs" pod "40092c87-92a3-48f7-8a2c-64f75215df63" (UID: "40092c87-92a3-48f7-8a2c-64f75215df63"). InnerVolumeSpecName "kube-api-access-t5trs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:21.442126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.442106 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5trs\" (UniqueName: \"kubernetes.io/projected/40092c87-92a3-48f7-8a2c-64f75215df63-kube-api-access-t5trs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.941256 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.941229 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" Oct 30 22:00:21.941597 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.941228 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z" event={"ID":"40092c87-92a3-48f7-8a2c-64f75215df63","Type":"ContainerDied","Data":"4abb7d4c5f416ac0365a048c17568e11ec327c14ea4524ebfd87058e2c79c8ad"} Oct 30 22:00:21.941597 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.941366 4335 scope.go:122] "RemoveContainer" containerID="08ef1ebc0e69c75c6e37ae737c23d624922be3384233d276e1358c218d9da675" Oct 30 22:00:21.943472 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.943442 4335 generic.go:356] "Generic (PLEG): container finished" podID="5809eb74-cd66-41b0-b527-fb8fcbc722ae" containerID="66c5dbf08cbb2691aa87a4923fc1470dd55e0b91c7df5fd81d966a706522ad44" exitCode=60 Oct 30 22:00:21.943590 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.943499 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" event={"ID":"5809eb74-cd66-41b0-b527-fb8fcbc722ae","Type":"ContainerDied","Data":"66c5dbf08cbb2691aa87a4923fc1470dd55e0b91c7df5fd81d966a706522ad44"} Oct 30 22:00:21.943857 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.943841 4335 scope.go:122] "RemoveContainer" containerID="66c5dbf08cbb2691aa87a4923fc1470dd55e0b91c7df5fd81d966a706522ad44" Oct 30 22:00:21.944093 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:00:21.944074 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9_odh-ai-gateway-infra(5809eb74-cd66-41b0-b527-fb8fcbc722ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" podUID="5809eb74-cd66-41b0-b527-fb8fcbc722ae" Oct 30 22:00:21.945669 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.945648 4335 generic.go:356] "Generic (PLEG): container finished" podID="79510ae6-8c22-45e7-98d2-d6174b001098" containerID="8eaf5fbe86bd559bc512a848e3fcabf22ffdc139cce6c13fab6f4e6fd3b42217" exitCode=60 Oct 30 22:00:21.945750 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.945682 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" event={"ID":"79510ae6-8c22-45e7-98d2-d6174b001098","Type":"ContainerDied","Data":"8eaf5fbe86bd559bc512a848e3fcabf22ffdc139cce6c13fab6f4e6fd3b42217"} Oct 30 22:00:21.945944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.945933 4335 scope.go:122] "RemoveContainer" containerID="8eaf5fbe86bd559bc512a848e3fcabf22ffdc139cce6c13fab6f4e6fd3b42217" Oct 30 22:00:21.946116 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:00:21.946100 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj_odh-ai-gateway-infra(79510ae6-8c22-45e7-98d2-d6174b001098)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" podUID="79510ae6-8c22-45e7-98d2-d6174b001098" Oct 30 22:00:21.970014 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.969997 4335 scope.go:122] "RemoveContainer" containerID="efef621e9a0fc44a94f05db9c3246bac747f47e5f1422678dfbbe70db783b582" Oct 30 22:00:21.982736 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:21.982720 4335 scope.go:122] "RemoveContainer" containerID="4146d2b15ef2cc5f0db48b656e699723f266408d4c67684b758f6003d024bd15" Oct 30 22:00:22.050862 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:22.050843 4335 scope.go:122] "RemoveContainer" containerID="0093266911ec86e9180e5f5d77f8b61c58e6dfee6426e8c391b12499a49d1a7d" Oct 30 22:00:22.051134 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:22.051119 4335 scope.go:122] "RemoveContainer" containerID="8530facd436971e2835c07f0c239ee11665af4c1fedad7ae9a33cc9830debe8a" Oct 30 22:00:22.073736 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:22.073700 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9"] Oct 30 22:00:22.352209 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:22.352174 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z"] Oct 30 22:00:22.357157 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:22.357129 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889960-59s4z"] Oct 30 22:00:22.955799 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:22.955768 4335 generic.go:356] "Generic (PLEG): container finished" podID="7753b052-e934-45b5-a8ca-b73f3fe0dddc" containerID="a62415872421fd706dc1263e9e2d702f2a2d2bb562c156dcc93cd883ca8ae17d" exitCode=60 Oct 30 22:00:22.956200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:22.955842 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" event={"ID":"7753b052-e934-45b5-a8ca-b73f3fe0dddc","Type":"ContainerDied","Data":"a62415872421fd706dc1263e9e2d702f2a2d2bb562c156dcc93cd883ca8ae17d"} Oct 30 22:00:22.956200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:22.955964 4335 scope.go:122] "RemoveContainer" containerID="0093266911ec86e9180e5f5d77f8b61c58e6dfee6426e8c391b12499a49d1a7d" Oct 30 22:00:22.956346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:22.956331 4335 scope.go:122] "RemoveContainer" containerID="a62415872421fd706dc1263e9e2d702f2a2d2bb562c156dcc93cd883ca8ae17d" Oct 30 22:00:22.956641 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:00:22.956598 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv_odh-ai-gateway-infra(7753b052-e934-45b5-a8ca-b73f3fe0dddc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" podUID="7753b052-e934-45b5-a8ca-b73f3fe0dddc" Oct 30 22:00:22.960216 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:22.960193 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" event={"ID":"d4221a80-5591-41ae-86e3-ff85b736b65e","Type":"ContainerStarted","Data":"d5f941c97ce3f1cbffb2acb5eb801b31c7f71ba8e1e4dd20804021f59c670820"} Oct 30 22:00:23.062011 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.061977 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40092c87-92a3-48f7-8a2c-64f75215df63" path="/var/lib/kubelet/pods/40092c87-92a3-48f7-8a2c-64f75215df63/volumes" Oct 30 22:00:23.115110 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.115089 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" Oct 30 22:00:23.121519 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.121496 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj"] Oct 30 22:00:23.258920 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.258892 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5809eb74-cd66-41b0-b527-fb8fcbc722ae-kube-api-access-tpt6g\" (UniqueName: \"kubernetes.io/projected/5809eb74-cd66-41b0-b527-fb8fcbc722ae-kube-api-access-tpt6g\") pod \"5809eb74-cd66-41b0-b527-fb8fcbc722ae\" (UID: \"5809eb74-cd66-41b0-b527-fb8fcbc722ae\") " Oct 30 22:00:23.259027 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.259013 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5809eb74-cd66-41b0-b527-fb8fcbc722ae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5809eb74-cd66-41b0-b527-fb8fcbc722ae-maas-api-service-ca\") pod \"5809eb74-cd66-41b0-b527-fb8fcbc722ae\" (UID: \"5809eb74-cd66-41b0-b527-fb8fcbc722ae\") " Oct 30 22:00:23.259425 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.259396 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5809eb74-cd66-41b0-b527-fb8fcbc722ae-maas-api-service-ca" pod "5809eb74-cd66-41b0-b527-fb8fcbc722ae" (UID: "5809eb74-cd66-41b0-b527-fb8fcbc722ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:23.260966 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.260947 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5809eb74-cd66-41b0-b527-fb8fcbc722ae-kube-api-access-tpt6g" pod "5809eb74-cd66-41b0-b527-fb8fcbc722ae" (UID: "5809eb74-cd66-41b0-b527-fb8fcbc722ae"). InnerVolumeSpecName "kube-api-access-tpt6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:23.270233 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.270216 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" Oct 30 22:00:23.360026 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.360002 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79510ae6-8c22-45e7-98d2-d6174b001098-kube-api-access-4j5wc\" (UniqueName: \"kubernetes.io/projected/79510ae6-8c22-45e7-98d2-d6174b001098-kube-api-access-4j5wc\") pod \"79510ae6-8c22-45e7-98d2-d6174b001098\" (UID: \"79510ae6-8c22-45e7-98d2-d6174b001098\") " Oct 30 22:00:23.360118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.360037 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79510ae6-8c22-45e7-98d2-d6174b001098-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79510ae6-8c22-45e7-98d2-d6174b001098-maas-api-service-ca\") pod \"79510ae6-8c22-45e7-98d2-d6174b001098\" (UID: \"79510ae6-8c22-45e7-98d2-d6174b001098\") " Oct 30 22:00:23.360446 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.360413 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79510ae6-8c22-45e7-98d2-d6174b001098-maas-api-service-ca" pod "79510ae6-8c22-45e7-98d2-d6174b001098" (UID: "79510ae6-8c22-45e7-98d2-d6174b001098"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:23.360486 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.360459 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5809eb74-cd66-41b0-b527-fb8fcbc722ae-maas-api-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:23.360549 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.360480 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpt6g\" (UniqueName: \"kubernetes.io/projected/5809eb74-cd66-41b0-b527-fb8fcbc722ae-kube-api-access-tpt6g\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:23.362188 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.362170 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79510ae6-8c22-45e7-98d2-d6174b001098-kube-api-access-4j5wc" pod "79510ae6-8c22-45e7-98d2-d6174b001098" (UID: "79510ae6-8c22-45e7-98d2-d6174b001098"). InnerVolumeSpecName "kube-api-access-4j5wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:23.461709 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.461605 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j5wc\" (UniqueName: \"kubernetes.io/projected/79510ae6-8c22-45e7-98d2-d6174b001098-kube-api-access-4j5wc\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:23.461709 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.461658 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79510ae6-8c22-45e7-98d2-d6174b001098-maas-api-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:23.973303 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.973273 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" Oct 30 22:00:23.973303 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.973282 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj" event={"ID":"79510ae6-8c22-45e7-98d2-d6174b001098","Type":"ContainerDied","Data":"bed7e4fb6a139a8cf85a36869fa2a3397c8177ce03924f998994b51ac16b1070"} Oct 30 22:00:23.973895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.973372 4335 scope.go:122] "RemoveContainer" containerID="8eaf5fbe86bd559bc512a848e3fcabf22ffdc139cce6c13fab6f4e6fd3b42217" Oct 30 22:00:23.975540 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.975518 4335 generic.go:356] "Generic (PLEG): container finished" podID="d4221a80-5591-41ae-86e3-ff85b736b65e" containerID="d5f941c97ce3f1cbffb2acb5eb801b31c7f71ba8e1e4dd20804021f59c670820" exitCode=60 Oct 30 22:00:23.975669 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.975602 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" event={"ID":"d4221a80-5591-41ae-86e3-ff85b736b65e","Type":"ContainerDied","Data":"d5f941c97ce3f1cbffb2acb5eb801b31c7f71ba8e1e4dd20804021f59c670820"} Oct 30 22:00:23.975944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.975928 4335 scope.go:122] "RemoveContainer" containerID="d5f941c97ce3f1cbffb2acb5eb801b31c7f71ba8e1e4dd20804021f59c670820" Oct 30 22:00:23.976180 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:00:23.976160 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-89nsj_odh-ai-gateway-infra(d4221a80-5591-41ae-86e3-ff85b736b65e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" podUID="d4221a80-5591-41ae-86e3-ff85b736b65e" Oct 30 22:00:23.977181 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.977158 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" event={"ID":"5809eb74-cd66-41b0-b527-fb8fcbc722ae","Type":"ContainerDied","Data":"e22de2910578c1625e8dc95cec7bf66478bb34f7d80f2787a18aa7da0254772c"} Oct 30 22:00:23.977272 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.977169 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9" Oct 30 22:00:23.984175 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.984154 4335 scope.go:122] "RemoveContainer" containerID="8530facd436971e2835c07f0c239ee11665af4c1fedad7ae9a33cc9830debe8a" Oct 30 22:00:23.996437 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:23.996421 4335 scope.go:122] "RemoveContainer" containerID="66c5dbf08cbb2691aa87a4923fc1470dd55e0b91c7df5fd81d966a706522ad44" Oct 30 22:00:24.085792 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:24.085767 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv"] Oct 30 22:00:24.253200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:24.253140 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj"] Oct 30 22:00:24.254051 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:24.254031 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" Oct 30 22:00:24.256764 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:24.256744 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889960-8m2cj"] Oct 30 22:00:24.316376 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:24.316352 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj"] Oct 30 22:00:24.368939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:24.368920 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9"] Oct 30 22:00:24.371318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:24.371298 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7753b052-e934-45b5-a8ca-b73f3fe0dddc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7753b052-e934-45b5-a8ca-b73f3fe0dddc-maas-api-service-ca\") pod \"7753b052-e934-45b5-a8ca-b73f3fe0dddc\" (UID: \"7753b052-e934-45b5-a8ca-b73f3fe0dddc\") " Oct 30 22:00:24.371451 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:24.371434 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7753b052-e934-45b5-a8ca-b73f3fe0dddc-kube-api-access-n7pcw\" (UniqueName: \"kubernetes.io/projected/7753b052-e934-45b5-a8ca-b73f3fe0dddc-kube-api-access-n7pcw\") pod \"7753b052-e934-45b5-a8ca-b73f3fe0dddc\" (UID: \"7753b052-e934-45b5-a8ca-b73f3fe0dddc\") " Oct 30 22:00:24.371740 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:24.371712 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7753b052-e934-45b5-a8ca-b73f3fe0dddc-maas-api-service-ca" pod "7753b052-e934-45b5-a8ca-b73f3fe0dddc" (UID: "7753b052-e934-45b5-a8ca-b73f3fe0dddc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:24.372898 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:24.372874 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7753b052-e934-45b5-a8ca-b73f3fe0dddc-maas-api-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:24.373679 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:24.373662 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7753b052-e934-45b5-a8ca-b73f3fe0dddc-kube-api-access-n7pcw" pod "7753b052-e934-45b5-a8ca-b73f3fe0dddc" (UID: "7753b052-e934-45b5-a8ca-b73f3fe0dddc"). InnerVolumeSpecName "kube-api-access-n7pcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:24.374718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:24.374700 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9fbe2-29889960-qjtv9"] Oct 30 22:00:24.498388 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:24.498302 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7pcw\" (UniqueName: \"kubernetes.io/projected/7753b052-e934-45b5-a8ca-b73f3fe0dddc-kube-api-access-n7pcw\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:24.988892 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:24.988857 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" event={"ID":"7753b052-e934-45b5-a8ca-b73f3fe0dddc","Type":"ContainerDied","Data":"9ca9207eb5cc4a1c2ca0d270185f69b5a46319f6624057abf6e6df2e9c57192d"} Oct 30 22:00:24.988892 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:24.988882 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv" Oct 30 22:00:24.989271 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:24.988930 4335 scope.go:122] "RemoveContainer" containerID="a62415872421fd706dc1263e9e2d702f2a2d2bb562c156dcc93cd883ca8ae17d" Oct 30 22:00:25.061372 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:25.061346 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5809eb74-cd66-41b0-b527-fb8fcbc722ae" path="/var/lib/kubelet/pods/5809eb74-cd66-41b0-b527-fb8fcbc722ae/volumes" Oct 30 22:00:25.061972 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:25.061946 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79510ae6-8c22-45e7-98d2-d6174b001098" path="/var/lib/kubelet/pods/79510ae6-8c22-45e7-98d2-d6174b001098/volumes" Oct 30 22:00:25.143794 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:25.143770 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" Oct 30 22:00:25.213234 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:25.213211 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv"] Oct 30 22:00:25.219291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:25.219271 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b2f165-29889960-6r5wv"] Oct 30 22:00:25.311714 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:25.311686 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4221a80-5591-41ae-86e3-ff85b736b65e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4221a80-5591-41ae-86e3-ff85b736b65e-maas-api-service-ca\") pod \"d4221a80-5591-41ae-86e3-ff85b736b65e\" (UID: \"d4221a80-5591-41ae-86e3-ff85b736b65e\") " Oct 30 22:00:25.311848 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:25.311801 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4221a80-5591-41ae-86e3-ff85b736b65e-kube-api-access-qlvjm\" (UniqueName: \"kubernetes.io/projected/d4221a80-5591-41ae-86e3-ff85b736b65e-kube-api-access-qlvjm\") pod \"d4221a80-5591-41ae-86e3-ff85b736b65e\" (UID: \"d4221a80-5591-41ae-86e3-ff85b736b65e\") " Oct 30 22:00:25.312014 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:25.311990 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4221a80-5591-41ae-86e3-ff85b736b65e-maas-api-service-ca" pod "d4221a80-5591-41ae-86e3-ff85b736b65e" (UID: "d4221a80-5591-41ae-86e3-ff85b736b65e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:25.312159 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:25.312144 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4221a80-5591-41ae-86e3-ff85b736b65e-maas-api-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:25.314068 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:25.314046 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4221a80-5591-41ae-86e3-ff85b736b65e-kube-api-access-qlvjm" pod "d4221a80-5591-41ae-86e3-ff85b736b65e" (UID: "d4221a80-5591-41ae-86e3-ff85b736b65e"). InnerVolumeSpecName "kube-api-access-qlvjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:25.319481 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:00:25.319458 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:00:25.412763 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:25.412732 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlvjm\" (UniqueName: \"kubernetes.io/projected/d4221a80-5591-41ae-86e3-ff85b736b65e-kube-api-access-qlvjm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:25.998552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:25.998530 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" Oct 30 22:00:25.998552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:25.998533 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj" event={"ID":"d4221a80-5591-41ae-86e3-ff85b736b65e","Type":"ContainerDied","Data":"1054c191ff07060480501dadf3626dc5d8c0d3d480b5d64d84c62c21fc96cafd"} Oct 30 22:00:25.999109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:25.998637 4335 scope.go:122] "RemoveContainer" containerID="d5f941c97ce3f1cbffb2acb5eb801b31c7f71ba8e1e4dd20804021f59c670820" Oct 30 22:00:26.165689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:26.165665 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj"] Oct 30 22:00:26.214543 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:26.214519 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-89nsj"] Oct 30 22:00:26.849507 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:26.849444 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:26.862670 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:26.862650 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:27.020677 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:27.020649 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:00:27.057082 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:27.057053 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7753b052-e934-45b5-a8ca-b73f3fe0dddc" path="/var/lib/kubelet/pods/7753b052-e934-45b5-a8ca-b73f3fe0dddc/volumes" Oct 30 22:00:27.057440 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:27.057426 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4221a80-5591-41ae-86e3-ff85b736b65e" path="/var/lib/kubelet/pods/d4221a80-5591-41ae-86e3-ff85b736b65e/volumes" Oct 30 22:00:32.329432 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:32.329391 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:00:37.926668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.926640 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr"] Oct 30 22:00:37.927130 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927117 4335 cpu_manager.go:414] "containerMap: removing container" podUID="d4221a80-5591-41ae-86e3-ff85b736b65e" containerName="cleanup" Oct 30 22:00:37.927170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927130 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4221a80-5591-41ae-86e3-ff85b736b65e" containerName="cleanup" Oct 30 22:00:37.927170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927142 4335 cpu_manager.go:414] "containerMap: removing container" podUID="40092c87-92a3-48f7-8a2c-64f75215df63" containerName="cleanup" Oct 30 22:00:37.927170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927147 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40092c87-92a3-48f7-8a2c-64f75215df63" containerName="cleanup" Oct 30 22:00:37.927170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927156 4335 cpu_manager.go:414] "containerMap: removing container" podUID="d4221a80-5591-41ae-86e3-ff85b736b65e" containerName="cleanup" Oct 30 22:00:37.927170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927161 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4221a80-5591-41ae-86e3-ff85b736b65e" containerName="cleanup" Oct 30 22:00:37.927170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927168 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5809eb74-cd66-41b0-b527-fb8fcbc722ae" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927173 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5809eb74-cd66-41b0-b527-fb8fcbc722ae" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927180 4335 cpu_manager.go:414] "containerMap: removing container" podUID="d4221a80-5591-41ae-86e3-ff85b736b65e" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927187 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4221a80-5591-41ae-86e3-ff85b736b65e" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927208 4335 cpu_manager.go:414] "containerMap: removing container" podUID="7753b052-e934-45b5-a8ca-b73f3fe0dddc" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927215 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7753b052-e934-45b5-a8ca-b73f3fe0dddc" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927224 4335 cpu_manager.go:414] "containerMap: removing container" podUID="40092c87-92a3-48f7-8a2c-64f75215df63" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927229 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40092c87-92a3-48f7-8a2c-64f75215df63" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927237 4335 cpu_manager.go:414] "containerMap: removing container" podUID="79510ae6-8c22-45e7-98d2-d6174b001098" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927241 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79510ae6-8c22-45e7-98d2-d6174b001098" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927252 4335 cpu_manager.go:414] "containerMap: removing container" podUID="79510ae6-8c22-45e7-98d2-d6174b001098" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927256 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79510ae6-8c22-45e7-98d2-d6174b001098" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927271 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5809eb74-cd66-41b0-b527-fb8fcbc722ae" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927275 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5809eb74-cd66-41b0-b527-fb8fcbc722ae" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927286 4335 cpu_manager.go:414] "containerMap: removing container" podUID="7753b052-e934-45b5-a8ca-b73f3fe0dddc" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927290 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7753b052-e934-45b5-a8ca-b73f3fe0dddc" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927367 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="7753b052-e934-45b5-a8ca-b73f3fe0dddc" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927374 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="40092c87-92a3-48f7-8a2c-64f75215df63" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927380 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="79510ae6-8c22-45e7-98d2-d6174b001098" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927387 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5809eb74-cd66-41b0-b527-fb8fcbc722ae" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927392 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="79510ae6-8c22-45e7-98d2-d6174b001098" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927399 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="7753b052-e934-45b5-a8ca-b73f3fe0dddc" containerName="cleanup" Oct 30 22:00:37.927394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927404 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5809eb74-cd66-41b0-b527-fb8fcbc722ae" containerName="cleanup" Oct 30 22:00:37.928038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927410 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="7753b052-e934-45b5-a8ca-b73f3fe0dddc" containerName="cleanup" Oct 30 22:00:37.928038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927415 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4221a80-5591-41ae-86e3-ff85b736b65e" containerName="cleanup" Oct 30 22:00:37.928038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927421 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="40092c87-92a3-48f7-8a2c-64f75215df63" containerName="cleanup" Oct 30 22:00:37.928038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927425 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4221a80-5591-41ae-86e3-ff85b736b65e" containerName="cleanup" Oct 30 22:00:37.928038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.927430 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4221a80-5591-41ae-86e3-ff85b736b65e" containerName="cleanup" Oct 30 22:00:37.934812 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.934795 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:37.988954 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:37.988925 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr"] Oct 30 22:00:38.034450 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.034422 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cef074\"/\"kube-root-ca.crt\"" Oct 30 22:00:38.034450 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.034438 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cef074\"/\"openshift-service-ca.crt\"" Oct 30 22:00:38.036421 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.036397 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cef074\"/\"simulator-w3-cef074-kserve-self-signed-certs\"" Oct 30 22:00:38.036557 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.036465 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cef074\"/\"default-dockercfg-4nxhw\"" Oct 30 22:00:38.058661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.058637 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8trz7\" (UniqueName: \"kubernetes.io/projected/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-kube-api-access-8trz7\") pod \"simulator-w3-cef074-kserve-57fdd9dd49-kkmgr\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.058784 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.058692 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-dshm\") pod \"simulator-w3-cef074-kserve-57fdd9dd49-kkmgr\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.058784 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.058709 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-model-cache\") pod \"simulator-w3-cef074-kserve-57fdd9dd49-kkmgr\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.058784 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.058728 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-tls-certs\") pod \"simulator-w3-cef074-kserve-57fdd9dd49-kkmgr\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.058918 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.058899 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-tmp-dir\") pod \"simulator-w3-cef074-kserve-57fdd9dd49-kkmgr\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.058954 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.058927 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-home\") pod \"simulator-w3-cef074-kserve-57fdd9dd49-kkmgr\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.160261 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.160215 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-dshm\") pod \"simulator-w3-cef074-kserve-57fdd9dd49-kkmgr\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.160261 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.160259 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-model-cache\") pod \"simulator-w3-cef074-kserve-57fdd9dd49-kkmgr\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.160261 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.160278 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-tls-certs\") pod \"simulator-w3-cef074-kserve-57fdd9dd49-kkmgr\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.160525 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.160399 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-tmp-dir\") pod \"simulator-w3-cef074-kserve-57fdd9dd49-kkmgr\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.160525 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.160424 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-home\") pod \"simulator-w3-cef074-kserve-57fdd9dd49-kkmgr\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.160525 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.160488 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8trz7\" (UniqueName: \"kubernetes.io/projected/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-kube-api-access-8trz7\") pod \"simulator-w3-cef074-kserve-57fdd9dd49-kkmgr\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.160800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.160765 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-model-cache\") pod \"simulator-w3-cef074-kserve-57fdd9dd49-kkmgr\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.160800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.160796 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-tmp-dir\") pod \"simulator-w3-cef074-kserve-57fdd9dd49-kkmgr\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.160967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.160937 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-home\") pod \"simulator-w3-cef074-kserve-57fdd9dd49-kkmgr\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.162691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.162673 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-dshm\") pod \"simulator-w3-cef074-kserve-57fdd9dd49-kkmgr\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.162919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.162904 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-tls-certs\") pod \"simulator-w3-cef074-kserve-57fdd9dd49-kkmgr\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.221056 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.221001 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8trz7\" (UniqueName: \"kubernetes.io/projected/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-kube-api-access-8trz7\") pod \"simulator-w3-cef074-kserve-57fdd9dd49-kkmgr\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.248017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.247996 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:38.634834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:38.634806 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr"] Oct 30 22:00:38.635145 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:00:38.635123 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda03a0c73_ecb0_4b79_a8bc_3feaeebe16c5.slice/crio-67e26428ec3ed711b3a82717bca06312d1f7af2ea20117b7f97df1b520bd645a WatchSource:0}: Error finding container 67e26428ec3ed711b3a82717bca06312d1f7af2ea20117b7f97df1b520bd645a: Status 404 returned error can't find the container with id 67e26428ec3ed711b3a82717bca06312d1f7af2ea20117b7f97df1b520bd645a Oct 30 22:00:39.100427 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:39.100388 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" event={"ID":"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5","Type":"ContainerStarted","Data":"3e0e57c529d7f9d2c94b9bc88fa59fd02f35b9dc68c56e27b1688270be91c23b"} Oct 30 22:00:39.100427 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:39.100430 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" event={"ID":"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5","Type":"ContainerStarted","Data":"67e26428ec3ed711b3a82717bca06312d1f7af2ea20117b7f97df1b520bd645a"} Oct 30 22:00:39.221069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:39.221020 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podStartSLOduration=2.221006807 podStartE2EDuration="2.221006807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:39.21787404 +0000 UTC m=+972.318305771" watchObservedRunningTime="2026-10-30 22:00:39.221006807 +0000 UTC m=+972.321438539" Oct 30 22:00:39.248232 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:39.248202 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:43.701104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.701072 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh"] Oct 30 22:00:43.701610 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.701596 4335 cpu_manager.go:414] "containerMap: removing container" podUID="7753b052-e934-45b5-a8ca-b73f3fe0dddc" containerName="cleanup" Oct 30 22:00:43.701610 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.701625 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7753b052-e934-45b5-a8ca-b73f3fe0dddc" containerName="cleanup" Oct 30 22:00:43.701724 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.701637 4335 cpu_manager.go:414] "containerMap: removing container" podUID="40092c87-92a3-48f7-8a2c-64f75215df63" containerName="cleanup" Oct 30 22:00:43.701724 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.701642 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40092c87-92a3-48f7-8a2c-64f75215df63" containerName="cleanup" Oct 30 22:00:43.701724 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.701656 4335 cpu_manager.go:414] "containerMap: removing container" podUID="79510ae6-8c22-45e7-98d2-d6174b001098" containerName="cleanup" Oct 30 22:00:43.701724 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.701661 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79510ae6-8c22-45e7-98d2-d6174b001098" containerName="cleanup" Oct 30 22:00:43.701724 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.701677 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5809eb74-cd66-41b0-b527-fb8fcbc722ae" containerName="cleanup" Oct 30 22:00:43.701724 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.701681 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5809eb74-cd66-41b0-b527-fb8fcbc722ae" containerName="cleanup" Oct 30 22:00:43.701902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.701764 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="40092c87-92a3-48f7-8a2c-64f75215df63" containerName="cleanup" Oct 30 22:00:43.701902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.701772 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5809eb74-cd66-41b0-b527-fb8fcbc722ae" containerName="cleanup" Oct 30 22:00:43.701902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.701778 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="79510ae6-8c22-45e7-98d2-d6174b001098" containerName="cleanup" Oct 30 22:00:43.704983 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.704969 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:43.763528 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.763499 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh"] Oct 30 22:00:43.809060 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.809035 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7df73b\"/\"kube-root-ca.crt\"" Oct 30 22:00:43.809218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.809200 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7df73b\"/\"openshift-service-ca.crt\"" Oct 30 22:00:43.809272 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.809228 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7df73b\"/\"default-dockercfg-l7s95\"" Oct 30 22:00:43.809272 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.809250 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7df73b\"/\"simulator-w0-7df73b-kserve-self-signed-certs\"" Oct 30 22:00:43.817649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.817628 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqjxw\" (UniqueName: \"kubernetes.io/projected/181e8750-1ec6-4564-a9d3-390e703dce76-kube-api-access-cqjxw\") pod \"simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:43.817745 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.817704 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-dshm\") pod \"simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:43.817813 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.817744 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-tmp-dir\") pod \"simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:43.817895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.817876 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/181e8750-1ec6-4564-a9d3-390e703dce76-tls-certs\") pod \"simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:43.817949 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.817915 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-model-cache\") pod \"simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:43.818040 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.818022 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-home\") pod \"simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:43.919561 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.919520 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-model-cache\") pod \"simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:43.919782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.919635 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-home\") pod \"simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:43.919782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.919751 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqjxw\" (UniqueName: \"kubernetes.io/projected/181e8750-1ec6-4564-a9d3-390e703dce76-kube-api-access-cqjxw\") pod \"simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:43.919918 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.919814 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-dshm\") pod \"simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:43.919918 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.919888 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-tmp-dir\") pod \"simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:43.920084 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.919975 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/181e8750-1ec6-4564-a9d3-390e703dce76-tls-certs\") pod \"simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:43.923710 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.923687 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/181e8750-1ec6-4564-a9d3-390e703dce76-tls-certs\") pod \"simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:43.923880 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.923716 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-home\") pod \"simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:43.923975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.923955 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-model-cache\") pod \"simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:43.926663 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.926587 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-tmp-dir\") pod \"simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:43.927922 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.927898 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-dshm\") pod \"simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:43.985797 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:43.985714 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqjxw\" (UniqueName: \"kubernetes.io/projected/181e8750-1ec6-4564-a9d3-390e703dce76-kube-api-access-cqjxw\") pod \"simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:44.019046 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:44.019022 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:44.399378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:44.399357 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh"] Oct 30 22:00:44.400509 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:00:44.400484 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod181e8750_1ec6_4564_a9d3_390e703dce76.slice/crio-644de6683fc4ecf3306fe3ab12db6800081cb67e7fd443ea9931cf4fcbefe975 WatchSource:0}: Error finding container 644de6683fc4ecf3306fe3ab12db6800081cb67e7fd443ea9931cf4fcbefe975: Status 404 returned error can't find the container with id 644de6683fc4ecf3306fe3ab12db6800081cb67e7fd443ea9931cf4fcbefe975 Oct 30 22:00:45.140479 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:45.140444 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" event={"ID":"181e8750-1ec6-4564-a9d3-390e703dce76","Type":"ContainerStarted","Data":"c28e2eef446723994ba4dbca5acc6555287ac7319b5cecd09e65dcc32fd18de2"} Oct 30 22:00:45.140479 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:45.140485 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" event={"ID":"181e8750-1ec6-4564-a9d3-390e703dce76","Type":"ContainerStarted","Data":"644de6683fc4ecf3306fe3ab12db6800081cb67e7fd443ea9931cf4fcbefe975"} Oct 30 22:00:45.258449 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:45.258402 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podStartSLOduration=2.258389205 podStartE2EDuration="2.258389205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:45.256938993 +0000 UTC m=+978.357370722" watchObservedRunningTime="2026-10-30 22:00:45.258389205 +0000 UTC m=+978.358820936" Oct 30 22:00:46.019712 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:46.019682 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:48.248774 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:48.248741 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:48.261649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:48.261598 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:49.178483 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:49.178454 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:00:54.020179 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:54.020147 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:54.032344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:54.032322 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:54.221144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:54.221118 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:00:55.322744 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:00:55.322708 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:00:56.132759 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:00:56.132724 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:01:25.326318 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:01:25.326274 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:01:26.534444 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:26.534414 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:01:27.911338 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:27.911310 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68"] Oct 30 22:01:27.915495 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:27.915473 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.020123 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.020098 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-b9fbe2\"/\"kube-root-ca.crt\"" Oct 30 22:01:28.020298 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.020175 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b9fbe2\"/\"default-dockercfg-l2zhl\"" Oct 30 22:01:28.020373 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.020235 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-b9fbe2\"/\"openshift-service-ca.crt\"" Oct 30 22:01:28.020373 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.020284 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b9fbe2\"/\"premium-w6-b9fbe2-kserve-self-signed-certs\"" Oct 30 22:01:28.025323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.025299 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68"] Oct 30 22:01:28.088855 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.088806 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-model-cache\") pod \"premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.089070 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.088871 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-home\") pod \"premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.089070 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.088911 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-tmp-dir\") pod \"premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.089070 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.088966 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-dshm\") pod \"premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.089070 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.089057 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6gmw\" (UniqueName: \"kubernetes.io/projected/9975ba00-5118-4e3f-984b-c6621e3c4f77-kube-api-access-q6gmw\") pod \"premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.089285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.089174 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9975ba00-5118-4e3f-984b-c6621e3c4f77-tls-certs\") pod \"premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.190745 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.190663 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6gmw\" (UniqueName: \"kubernetes.io/projected/9975ba00-5118-4e3f-984b-c6621e3c4f77-kube-api-access-q6gmw\") pod \"premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.190902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.190811 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9975ba00-5118-4e3f-984b-c6621e3c4f77-tls-certs\") pod \"premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.190902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.190854 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-model-cache\") pod \"premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.191044 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.190909 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-home\") pod \"premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.191044 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.190942 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-tmp-dir\") pod \"premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.191156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.191037 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-dshm\") pod \"premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.191473 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.191446 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-tmp-dir\") pod \"premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.191591 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.191497 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-model-cache\") pod \"premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.191591 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.191524 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-home\") pod \"premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.193363 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.193342 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-dshm\") pod \"premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.193761 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.193740 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9975ba00-5118-4e3f-984b-c6621e3c4f77-tls-certs\") pod \"premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.254500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.254471 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6gmw\" (UniqueName: \"kubernetes.io/projected/9975ba00-5118-4e3f-984b-c6621e3c4f77-kube-api-access-q6gmw\") pod \"premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.527250 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.527168 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:28.907589 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:28.907566 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68"] Oct 30 22:01:28.908058 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:01:28.908035 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9975ba00_5118_4e3f_984b_c6621e3c4f77.slice/crio-78e3ea3fcd899cabf467f7abee3f1931cf4312c71a3e7a2713ddff03931e6dcc WatchSource:0}: Error finding container 78e3ea3fcd899cabf467f7abee3f1931cf4312c71a3e7a2713ddff03931e6dcc: Status 404 returned error can't find the container with id 78e3ea3fcd899cabf467f7abee3f1931cf4312c71a3e7a2713ddff03931e6dcc Oct 30 22:01:29.480249 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:29.480216 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" event={"ID":"9975ba00-5118-4e3f-984b-c6621e3c4f77","Type":"ContainerStarted","Data":"653777e7949acc4d0a115d3dcd131cda3c5b0748b43019ea39de612004351bb8"} Oct 30 22:01:29.480249 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:29.480257 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" event={"ID":"9975ba00-5118-4e3f-984b-c6621e3c4f77","Type":"ContainerStarted","Data":"78e3ea3fcd899cabf467f7abee3f1931cf4312c71a3e7a2713ddff03931e6dcc"} Oct 30 22:01:29.528060 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:29.528034 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:29.598366 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:29.598323 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podStartSLOduration=2.59830845 podStartE2EDuration="2.59830845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:29.596000728 +0000 UTC m=+1022.696432460" watchObservedRunningTime="2026-10-30 22:01:29.59830845 +0000 UTC m=+1022.698740182" Oct 30 22:01:38.527410 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:38.527317 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:38.539495 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:38.539468 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:38.552464 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:38.552441 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:01:47.216984 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:47.216956 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:01:52.410065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.410025 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8"] Oct 30 22:01:52.414295 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.414272 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.469696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.469668 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-536975\"/\"premium-w4-536975-kserve-self-signed-certs\"" Oct 30 22:01:52.522442 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.522413 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8"] Oct 30 22:01:52.533480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.533455 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-dshm\") pod \"premium-w4-536975-kserve-844f65b8b9-c25r8\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.533599 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.533490 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l25mc\" (UniqueName: \"kubernetes.io/projected/43393e2f-feb9-4698-8f80-5291391625b3-kube-api-access-l25mc\") pod \"premium-w4-536975-kserve-844f65b8b9-c25r8\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.533599 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.533512 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-home\") pod \"premium-w4-536975-kserve-844f65b8b9-c25r8\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.533599 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.533578 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-tmp-dir\") pod \"premium-w4-536975-kserve-844f65b8b9-c25r8\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.533727 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.533680 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-model-cache\") pod \"premium-w4-536975-kserve-844f65b8b9-c25r8\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.533821 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.533806 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43393e2f-feb9-4698-8f80-5291391625b3-tls-certs\") pod \"premium-w4-536975-kserve-844f65b8b9-c25r8\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.634834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.634801 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-model-cache\") pod \"premium-w4-536975-kserve-844f65b8b9-c25r8\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.635020 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.634945 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43393e2f-feb9-4698-8f80-5291391625b3-tls-certs\") pod \"premium-w4-536975-kserve-844f65b8b9-c25r8\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.635089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.635018 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-dshm\") pod \"premium-w4-536975-kserve-844f65b8b9-c25r8\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.635089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.635057 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l25mc\" (UniqueName: \"kubernetes.io/projected/43393e2f-feb9-4698-8f80-5291391625b3-kube-api-access-l25mc\") pod \"premium-w4-536975-kserve-844f65b8b9-c25r8\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.635198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.635093 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-home\") pod \"premium-w4-536975-kserve-844f65b8b9-c25r8\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.635198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.635123 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-tmp-dir\") pod \"premium-w4-536975-kserve-844f65b8b9-c25r8\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.635338 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.635314 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-model-cache\") pod \"premium-w4-536975-kserve-844f65b8b9-c25r8\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.635513 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.635487 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-home\") pod \"premium-w4-536975-kserve-844f65b8b9-c25r8\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.635668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.635494 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-tmp-dir\") pod \"premium-w4-536975-kserve-844f65b8b9-c25r8\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.637477 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.637460 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-dshm\") pod \"premium-w4-536975-kserve-844f65b8b9-c25r8\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.637773 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.637750 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43393e2f-feb9-4698-8f80-5291391625b3-tls-certs\") pod \"premium-w4-536975-kserve-844f65b8b9-c25r8\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.702940 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.702881 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l25mc\" (UniqueName: \"kubernetes.io/projected/43393e2f-feb9-4698-8f80-5291391625b3-kube-api-access-l25mc\") pod \"premium-w4-536975-kserve-844f65b8b9-c25r8\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:52.726785 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:52.726763 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:53.116386 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:01:53.116360 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43393e2f_feb9_4698_8f80_5291391625b3.slice/crio-0b6f70bc2456cda11a204a54a962bfae85ff3f0c700ed8cbdf5d3725408141b7 WatchSource:0}: Error finding container 0b6f70bc2456cda11a204a54a962bfae85ff3f0c700ed8cbdf5d3725408141b7: Status 404 returned error can't find the container with id 0b6f70bc2456cda11a204a54a962bfae85ff3f0c700ed8cbdf5d3725408141b7 Oct 30 22:01:53.116638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:53.116599 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8"] Oct 30 22:01:53.649641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:53.649568 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" event={"ID":"43393e2f-feb9-4698-8f80-5291391625b3","Type":"ContainerStarted","Data":"2b2bee240b587aa4b71594af8427567f2e16e17d6315d1425b01bfa35d54f531"} Oct 30 22:01:53.650003 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:53.649657 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" event={"ID":"43393e2f-feb9-4698-8f80-5291391625b3","Type":"ContainerStarted","Data":"0b6f70bc2456cda11a204a54a962bfae85ff3f0c700ed8cbdf5d3725408141b7"} Oct 30 22:01:53.727135 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:53.727108 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:01:53.767113 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:01:53.767068 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podStartSLOduration=1.767054715 podStartE2EDuration="1.767054715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:53.766356644 +0000 UTC m=+1046.866788377" watchObservedRunningTime="2026-10-30 22:01:53.767054715 +0000 UTC m=+1046.867486447" Oct 30 22:01:55.321820 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:01:55.321779 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:02:02.726922 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:02.726887 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:02:02.739878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:02.739850 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:02:03.739890 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:03.739861 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:02:09.442475 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.442445 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk"] Oct 30 22:02:09.446595 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.446578 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:09.505499 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.505468 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk"] Oct 30 22:02:09.550033 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.550001 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cef074\"/\"premium-w3-cef074-kserve-self-signed-certs\"" Oct 30 22:02:09.623125 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.623093 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-tmp-dir\") pod \"premium-w3-cef074-kserve-6899757bb8-zx7hk\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:09.623300 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.623173 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74qh9\" (UniqueName: \"kubernetes.io/projected/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-kube-api-access-74qh9\") pod \"premium-w3-cef074-kserve-6899757bb8-zx7hk\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:09.623445 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.623362 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-tls-certs\") pod \"premium-w3-cef074-kserve-6899757bb8-zx7hk\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:09.623508 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.623471 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-home\") pod \"premium-w3-cef074-kserve-6899757bb8-zx7hk\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:09.623563 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.623514 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-model-cache\") pod \"premium-w3-cef074-kserve-6899757bb8-zx7hk\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:09.623648 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.623594 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-dshm\") pod \"premium-w3-cef074-kserve-6899757bb8-zx7hk\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:09.724949 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.724856 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-home\") pod \"premium-w3-cef074-kserve-6899757bb8-zx7hk\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:09.724949 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.724900 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-model-cache\") pod \"premium-w3-cef074-kserve-6899757bb8-zx7hk\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:09.724949 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.724939 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-dshm\") pod \"premium-w3-cef074-kserve-6899757bb8-zx7hk\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:09.725232 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.724979 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-tmp-dir\") pod \"premium-w3-cef074-kserve-6899757bb8-zx7hk\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:09.725232 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.725053 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74qh9\" (UniqueName: \"kubernetes.io/projected/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-kube-api-access-74qh9\") pod \"premium-w3-cef074-kserve-6899757bb8-zx7hk\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:09.725232 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.725153 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-tls-certs\") pod \"premium-w3-cef074-kserve-6899757bb8-zx7hk\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:09.725391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.725311 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-home\") pod \"premium-w3-cef074-kserve-6899757bb8-zx7hk\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:09.725391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.725361 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-model-cache\") pod \"premium-w3-cef074-kserve-6899757bb8-zx7hk\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:09.725724 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.725702 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-tmp-dir\") pod \"premium-w3-cef074-kserve-6899757bb8-zx7hk\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:09.727775 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.727745 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-tls-certs\") pod \"premium-w3-cef074-kserve-6899757bb8-zx7hk\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:09.727966 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.727946 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-dshm\") pod \"premium-w3-cef074-kserve-6899757bb8-zx7hk\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:09.786230 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:09.786198 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74qh9\" (UniqueName: \"kubernetes.io/projected/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-kube-api-access-74qh9\") pod \"premium-w3-cef074-kserve-6899757bb8-zx7hk\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:10.059011 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:10.058981 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:10.439820 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:10.439777 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk"] Oct 30 22:02:10.439983 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:02:10.439909 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc592bee9_4da3_43f3_9347_a8ee5dc1fce6.slice/crio-643f3e7d1110fde588c851dbb86a803b380a108df6734187a85c65874ad2d94e WatchSource:0}: Error finding container 643f3e7d1110fde588c851dbb86a803b380a108df6734187a85c65874ad2d94e: Status 404 returned error can't find the container with id 643f3e7d1110fde588c851dbb86a803b380a108df6734187a85c65874ad2d94e Oct 30 22:02:10.785643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:10.785554 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" event={"ID":"c592bee9-4da3-43f3-9347-a8ee5dc1fce6","Type":"ContainerStarted","Data":"b70397000885f2e87902708d6716e80ceee3b915224a6ef7189c5caa4fce504f"} Oct 30 22:02:10.785643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:10.785602 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" event={"ID":"c592bee9-4da3-43f3-9347-a8ee5dc1fce6","Type":"ContainerStarted","Data":"643f3e7d1110fde588c851dbb86a803b380a108df6734187a85c65874ad2d94e"} Oct 30 22:02:10.903942 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:10.903891 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podStartSLOduration=1.9038780960000001 podStartE2EDuration="1.903878096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:10.901971045 +0000 UTC m=+1064.002402778" watchObservedRunningTime="2026-10-30 22:02:10.903878096 +0000 UTC m=+1064.004309827" Oct 30 22:02:11.062074 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:11.062037 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:14.627782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:14.627748 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b"] Oct 30 22:02:14.632584 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:14.632564 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b" Oct 30 22:02:14.679177 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:14.679150 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee15a032-782e-4441-a3df-8352743ba9fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af36e54a-vbm2b\" (UID: \"ee15a032-782e-4441-a3df-8352743ba9fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b" Oct 30 22:02:14.679277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:14.679192 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw5q5\" (UniqueName: \"kubernetes.io/projected/ee15a032-782e-4441-a3df-8352743ba9fc-kube-api-access-gw5q5\") pod \"maas-api-revoke-sub-af36e54a-vbm2b\" (UID: \"ee15a032-782e-4441-a3df-8352743ba9fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b" Oct 30 22:02:14.679351 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:14.679333 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee15a032-782e-4441-a3df-8352743ba9fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af36e54a-vbm2b\" (UID: \"ee15a032-782e-4441-a3df-8352743ba9fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b" Oct 30 22:02:14.688318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:14.688295 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b"] Oct 30 22:02:14.732755 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:14.732712 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:14.732943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:14.732763 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:02:14.732943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:14.732898 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:02:14.733881 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:14.733864 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:02:14.780884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:14.780861 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee15a032-782e-4441-a3df-8352743ba9fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af36e54a-vbm2b\" (UID: \"ee15a032-782e-4441-a3df-8352743ba9fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b" Oct 30 22:02:14.780990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:14.780912 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw5q5\" (UniqueName: \"kubernetes.io/projected/ee15a032-782e-4441-a3df-8352743ba9fc-kube-api-access-gw5q5\") pod \"maas-api-revoke-sub-af36e54a-vbm2b\" (UID: \"ee15a032-782e-4441-a3df-8352743ba9fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b" Oct 30 22:02:14.781033 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:14.781010 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee15a032-782e-4441-a3df-8352743ba9fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af36e54a-vbm2b\" (UID: \"ee15a032-782e-4441-a3df-8352743ba9fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b" Oct 30 22:02:14.781454 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:14.781427 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee15a032-782e-4441-a3df-8352743ba9fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af36e54a-vbm2b\" (UID: \"ee15a032-782e-4441-a3df-8352743ba9fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b" Oct 30 22:02:14.783537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:14.783512 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee15a032-782e-4441-a3df-8352743ba9fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af36e54a-vbm2b\" (UID: \"ee15a032-782e-4441-a3df-8352743ba9fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b" Oct 30 22:02:14.842126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:14.842101 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw5q5\" (UniqueName: \"kubernetes.io/projected/ee15a032-782e-4441-a3df-8352743ba9fc-kube-api-access-gw5q5\") pod \"maas-api-revoke-sub-af36e54a-vbm2b\" (UID: \"ee15a032-782e-4441-a3df-8352743ba9fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b" Oct 30 22:02:14.950548 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:14.950485 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b" Oct 30 22:02:15.336359 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:15.336334 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b"] Oct 30 22:02:15.337454 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:02:15.337425 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee15a032_782e_4441_a3df_8352743ba9fc.slice/crio-5666aeeac61496a10a6ec4159478e4e5b3f5444c33293c8257baa80e2fbf6021 WatchSource:0}: Error finding container 5666aeeac61496a10a6ec4159478e4e5b3f5444c33293c8257baa80e2fbf6021: Status 404 returned error can't find the container with id 5666aeeac61496a10a6ec4159478e4e5b3f5444c33293c8257baa80e2fbf6021 Oct 30 22:02:15.824663 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:15.824630 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b" event={"ID":"ee15a032-782e-4441-a3df-8352743ba9fc","Type":"ContainerStarted","Data":"6720af95a7d4bda5af4d5d224319560ed3374469a79ab63e56e6e7cca3f275a7"} Oct 30 22:02:15.825135 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:15.824682 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b" event={"ID":"ee15a032-782e-4441-a3df-8352743ba9fc","Type":"ContainerStarted","Data":"5666aeeac61496a10a6ec4159478e4e5b3f5444c33293c8257baa80e2fbf6021"} Oct 30 22:02:15.943254 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:15.943165 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b" podStartSLOduration=1.943147827 podStartE2EDuration="1.943147827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:15.939405194 +0000 UTC m=+1069.039836927" watchObservedRunningTime="2026-10-30 22:02:15.943147827 +0000 UTC m=+1069.043579559" Oct 30 22:02:16.602207 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:16.602171 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:02:16.833819 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:16.833784 4335 generic.go:356] "Generic (PLEG): container finished" podID="ee15a032-782e-4441-a3df-8352743ba9fc" containerID="6720af95a7d4bda5af4d5d224319560ed3374469a79ab63e56e6e7cca3f275a7" exitCode=0 Oct 30 22:02:16.834246 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:16.833822 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b" event={"ID":"ee15a032-782e-4441-a3df-8352743ba9fc","Type":"ContainerDied","Data":"6720af95a7d4bda5af4d5d224319560ed3374469a79ab63e56e6e7cca3f275a7"} Oct 30 22:02:17.994279 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:17.994253 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b" Oct 30 22:02:18.120732 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:18.120639 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee15a032-782e-4441-a3df-8352743ba9fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee15a032-782e-4441-a3df-8352743ba9fc-maas-api-openshift-ca\") pod \"ee15a032-782e-4441-a3df-8352743ba9fc\" (UID: \"ee15a032-782e-4441-a3df-8352743ba9fc\") " Oct 30 22:02:18.120732 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:18.120716 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee15a032-782e-4441-a3df-8352743ba9fc-kube-api-access-gw5q5\" (UniqueName: \"kubernetes.io/projected/ee15a032-782e-4441-a3df-8352743ba9fc-kube-api-access-gw5q5\") pod \"ee15a032-782e-4441-a3df-8352743ba9fc\" (UID: \"ee15a032-782e-4441-a3df-8352743ba9fc\") " Oct 30 22:02:18.120941 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:18.120762 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee15a032-782e-4441-a3df-8352743ba9fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee15a032-782e-4441-a3df-8352743ba9fc-maas-api-xks-ca\") pod \"ee15a032-782e-4441-a3df-8352743ba9fc\" (UID: \"ee15a032-782e-4441-a3df-8352743ba9fc\") " Oct 30 22:02:18.121010 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:18.120952 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee15a032-782e-4441-a3df-8352743ba9fc-maas-api-openshift-ca" pod "ee15a032-782e-4441-a3df-8352743ba9fc" (UID: "ee15a032-782e-4441-a3df-8352743ba9fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:18.121223 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:18.121199 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee15a032-782e-4441-a3df-8352743ba9fc-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:18.123186 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:18.123155 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee15a032-782e-4441-a3df-8352743ba9fc-maas-api-xks-ca" pod "ee15a032-782e-4441-a3df-8352743ba9fc" (UID: "ee15a032-782e-4441-a3df-8352743ba9fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:18.123313 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:18.123289 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee15a032-782e-4441-a3df-8352743ba9fc-kube-api-access-gw5q5" pod "ee15a032-782e-4441-a3df-8352743ba9fc" (UID: "ee15a032-782e-4441-a3df-8352743ba9fc"). InnerVolumeSpecName "kube-api-access-gw5q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:18.221821 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:18.221797 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw5q5\" (UniqueName: \"kubernetes.io/projected/ee15a032-782e-4441-a3df-8352743ba9fc-kube-api-access-gw5q5\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:18.221821 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:18.221821 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee15a032-782e-4441-a3df-8352743ba9fc-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:18.851493 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:18.851462 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b" Oct 30 22:02:18.851493 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:18.851465 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b" event={"ID":"ee15a032-782e-4441-a3df-8352743ba9fc","Type":"ContainerDied","Data":"5666aeeac61496a10a6ec4159478e4e5b3f5444c33293c8257baa80e2fbf6021"} Oct 30 22:02:18.851725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:18.851504 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5666aeeac61496a10a6ec4159478e4e5b3f5444c33293c8257baa80e2fbf6021" Oct 30 22:02:20.059346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:20.059317 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:20.076465 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:20.076442 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:20.874416 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:20.874385 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:02:25.320513 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:02:25.320479 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:02:26.280673 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.280643 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw"] Oct 30 22:02:26.281187 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.281173 4335 cpu_manager.go:414] "containerMap: removing container" podUID="ee15a032-782e-4441-a3df-8352743ba9fc" containerName="revoke-keys" Oct 30 22:02:26.281250 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.281189 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee15a032-782e-4441-a3df-8352743ba9fc" containerName="revoke-keys" Oct 30 22:02:26.281286 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.281278 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee15a032-782e-4441-a3df-8352743ba9fc" containerName="revoke-keys" Oct 30 22:02:26.284730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.284715 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.336780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.336759 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-cef074-openshift-default-dockercfg-vkbnc\"" Oct 30 22:02:26.389874 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.389853 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw"] Oct 30 22:02:26.400723 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.400700 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-workload-certs\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.400841 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.400736 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-data\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.400841 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.400764 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-podinfo\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.400921 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.400850 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istiod-ca-cert\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.400921 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.400889 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-token\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.400996 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.400923 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-credential-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.400996 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.400967 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-envoy\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.401064 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.401052 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85rf7\" (UniqueName: \"kubernetes.io/projected/6c79ac4e-ddf1-41d0-a8ac-326974828e18-kube-api-access-85rf7\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.401100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.401089 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-workload-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.502085 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.502058 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85rf7\" (UniqueName: \"kubernetes.io/projected/6c79ac4e-ddf1-41d0-a8ac-326974828e18-kube-api-access-85rf7\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.502191 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.502100 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-workload-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.502191 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.502129 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-workload-certs\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.502191 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.502163 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-data\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.502368 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.502202 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-podinfo\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.502368 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.502231 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istiod-ca-cert\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.502467 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.502399 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-token\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.502830 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.502511 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-credential-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.502830 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.502532 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-workload-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.502830 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.502561 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-workload-certs\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.502830 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.502649 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-envoy\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.502830 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.502701 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-data\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.503152 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.502838 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-credential-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.503152 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.503072 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istiod-ca-cert\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.504968 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.504947 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-podinfo\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.505060 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.504949 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-envoy\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.562936 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.562912 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-token\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.563032 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.563011 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85rf7\" (UniqueName: \"kubernetes.io/projected/6c79ac4e-ddf1-41d0-a8ac-326974828e18-kube-api-access-85rf7\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-pw5rw\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:26.598597 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:26.598575 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:27.046992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:27.046921 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw"] Oct 30 22:02:27.051114 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:02:27.051085 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c79ac4e_ddf1_41d0_a8ac_326974828e18.slice/crio-c7df01b7402c7f79412d263c8f66674d9707d555f18e1972fd224d50be9e13dc WatchSource:0}: Error finding container c7df01b7402c7f79412d263c8f66674d9707d555f18e1972fd224d50be9e13dc: Status 404 returned error can't find the container with id c7df01b7402c7f79412d263c8f66674d9707d555f18e1972fd224d50be9e13dc Oct 30 22:02:27.053578 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:27.053545 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:27.053730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:27.053607 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:27.053730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:27.053673 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:27.927080 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:27.927040 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" event={"ID":"6c79ac4e-ddf1-41d0-a8ac-326974828e18","Type":"ContainerStarted","Data":"541c4f8f00dff5301f42a0547ccf95b560d883f3bf5cf167d72329ffe244d9a6"} Oct 30 22:02:27.927508 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:27.927134 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" event={"ID":"6c79ac4e-ddf1-41d0-a8ac-326974828e18","Type":"ContainerStarted","Data":"c7df01b7402c7f79412d263c8f66674d9707d555f18e1972fd224d50be9e13dc"} Oct 30 22:02:28.060982 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:28.060936 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" podStartSLOduration=2.060922418 podStartE2EDuration="2.060922418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:28.047193535 +0000 UTC m=+1081.147625267" watchObservedRunningTime="2026-10-30 22:02:28.060922418 +0000 UTC m=+1081.161354149" Oct 30 22:02:28.603640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:28.601854 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:28.620653 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:28.619699 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:02:28.620653 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:28.619756 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" podUID="6c79ac4e-ddf1-41d0-a8ac-326974828e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 22:02:29.184844 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:29.184800 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf"] Oct 30 22:02:29.185277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:29.185052 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" podUID="d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" containerName="istio-proxy" containerID="cri-o://0e5b2586ce5c1bf9d8a0697c2f9419fc88598a997f998c3ea14dfd1383aed9c1" gracePeriod=30 Oct 30 22:02:29.233649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:29.233060 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg"] Oct 30 22:02:29.233649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:29.233293 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" podUID="91fba568-9213-4aa9-a093-17df3f0c506e" containerName="istio-proxy" containerID="cri-o://3eb1c251b5a4ccbd3adf3a07e097d1a7388b40950d17b3cc5eafdd045a984d88" gracePeriod=30 Oct 30 22:02:29.237543 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:29.237516 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv"] Oct 30 22:02:29.237981 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:29.237953 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" podUID="ba8a2354-02ca-46cf-bea7-d97781a505f5" containerName="istio-proxy" containerID="cri-o://f30deb5e197025c9592cdcc8cb863bb149cef7a08645591e2ef2a9d16fce6e14" gracePeriod=30 Oct 30 22:02:29.600394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:29.599966 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:02:29.600394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:29.600026 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" podUID="6c79ac4e-ddf1-41d0-a8ac-326974828e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 22:02:30.605641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:30.601969 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:02:30.605641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:30.602039 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" podUID="6c79ac4e-ddf1-41d0-a8ac-326974828e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 22:02:31.606641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:31.601949 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:02:31.606641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:31.602021 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" podUID="6c79ac4e-ddf1-41d0-a8ac-326974828e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 22:02:32.600641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:32.600535 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:02:32.600641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:32.600606 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" podUID="6c79ac4e-ddf1-41d0-a8ac-326974828e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 22:02:33.774560 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:33.774524 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:33.775096 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:33.775016 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:33.776154 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:33.776132 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:02:34.539507 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.538767 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 22:02:34.594093 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.594071 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 22:02:34.597112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.597094 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 22:02:34.600405 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.600387 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-workload-socket\") pod \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " Oct 30 22:02:34.600517 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.600502 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-kube-api-access-pnjzk\" (UniqueName: \"kubernetes.io/projected/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-kube-api-access-pnjzk\") pod \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " Oct 30 22:02:34.600583 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.600531 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-workload-certs\") pod \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " Oct 30 22:02:34.600583 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.600547 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-data\") pod \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " Oct 30 22:02:34.600726 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.600607 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-podinfo\") pod \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " Oct 30 22:02:34.600726 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.600683 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-workload-socket" pod "d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" (UID: "d1f1ef3d-7992-4f3c-8064-3f3c23bb6481"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:34.600837 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.600739 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istiod-ca-cert\") pod \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " Oct 30 22:02:34.600837 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.600795 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-credential-socket\") pod \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " Oct 30 22:02:34.600946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.600846 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-token\" (UniqueName: \"kubernetes.io/projected/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-token\") pod \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " Oct 30 22:02:34.600946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.600858 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-workload-certs" pod "d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" (UID: "d1f1ef3d-7992-4f3c-8064-3f3c23bb6481"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:34.600946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.600886 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-envoy\") pod \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\" (UID: \"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481\") " Oct 30 22:02:34.600946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.600887 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-data" pod "d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" (UID: "d1f1ef3d-7992-4f3c-8064-3f3c23bb6481"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:34.601170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.601064 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-credential-socket" pod "d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" (UID: "d1f1ef3d-7992-4f3c-8064-3f3c23bb6481"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:34.601228 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.601203 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istiod-ca-cert" pod "d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" (UID: "d1f1ef3d-7992-4f3c-8064-3f3c23bb6481"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:34.601414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.601382 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.601414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.601407 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.601732 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.601423 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.601732 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.601437 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.601732 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.601450 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.603245 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.603217 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-podinfo" pod "d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" (UID: "d1f1ef3d-7992-4f3c-8064-3f3c23bb6481"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:34.603506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.603488 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-kube-api-access-pnjzk" pod "d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" (UID: "d1f1ef3d-7992-4f3c-8064-3f3c23bb6481"). InnerVolumeSpecName "kube-api-access-pnjzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:34.603681 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.603660 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-envoy" pod "d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" (UID: "d1f1ef3d-7992-4f3c-8064-3f3c23bb6481"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:34.603750 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.603659 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-token" pod "d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" (UID: "d1f1ef3d-7992-4f3c-8064-3f3c23bb6481"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:34.702358 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.702329 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-token\" (UniqueName: \"kubernetes.io/projected/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-token\") pod \"ba8a2354-02ca-46cf-bea7-d97781a505f5\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " Oct 30 22:02:34.702506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.702377 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-workload-socket\") pod \"91fba568-9213-4aa9-a093-17df3f0c506e\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " Oct 30 22:02:34.702506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.702402 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-istio-data\") pod \"91fba568-9213-4aa9-a093-17df3f0c506e\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " Oct 30 22:02:34.702506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.702442 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91fba568-9213-4aa9-a093-17df3f0c506e-kube-api-access-6nmq5\" (UniqueName: \"kubernetes.io/projected/91fba568-9213-4aa9-a093-17df3f0c506e-kube-api-access-6nmq5\") pod \"91fba568-9213-4aa9-a093-17df3f0c506e\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " Oct 30 22:02:34.702506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.702477 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba8a2354-02ca-46cf-bea7-d97781a505f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8a2354-02ca-46cf-bea7-d97781a505f5-istiod-ca-cert\") pod \"ba8a2354-02ca-46cf-bea7-d97781a505f5\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " Oct 30 22:02:34.702765 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.702552 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-istio-envoy\") pod \"91fba568-9213-4aa9-a093-17df3f0c506e\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " Oct 30 22:02:34.702765 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.702586 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-data\") pod \"ba8a2354-02ca-46cf-bea7-d97781a505f5\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " Oct 30 22:02:34.702765 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.702643 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-credential-socket\") pod \"ba8a2354-02ca-46cf-bea7-d97781a505f5\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " Oct 30 22:02:34.702765 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.702679 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91fba568-9213-4aa9-a093-17df3f0c506e-istio-token\" (UniqueName: \"kubernetes.io/projected/91fba568-9213-4aa9-a093-17df3f0c506e-istio-token\") pod \"91fba568-9213-4aa9-a093-17df3f0c506e\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " Oct 30 22:02:34.702765 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.702706 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91fba568-9213-4aa9-a093-17df3f0c506e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fba568-9213-4aa9-a093-17df3f0c506e-istiod-ca-cert\") pod \"91fba568-9213-4aa9-a093-17df3f0c506e\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " Oct 30 22:02:34.702980 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.702764 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-workload-certs\") pod \"91fba568-9213-4aa9-a093-17df3f0c506e\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " Oct 30 22:02:34.703030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.702678 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-workload-socket" pod "91fba568-9213-4aa9-a093-17df3f0c506e" (UID: "91fba568-9213-4aa9-a093-17df3f0c506e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:34.703030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.702743 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-istio-data" pod "91fba568-9213-4aa9-a093-17df3f0c506e" (UID: "91fba568-9213-4aa9-a093-17df3f0c506e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:34.703030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.702928 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-credential-socket" pod "ba8a2354-02ca-46cf-bea7-d97781a505f5" (UID: "ba8a2354-02ca-46cf-bea7-d97781a505f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:34.703030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.702943 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-data" pod "ba8a2354-02ca-46cf-bea7-d97781a505f5" (UID: "ba8a2354-02ca-46cf-bea7-d97781a505f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:34.703169 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.703052 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-workload-socket\") pod \"ba8a2354-02ca-46cf-bea7-d97781a505f5\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " Oct 30 22:02:34.703169 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.703086 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91fba568-9213-4aa9-a093-17df3f0c506e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fba568-9213-4aa9-a093-17df3f0c506e-istio-podinfo\") pod \"91fba568-9213-4aa9-a093-17df3f0c506e\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " Oct 30 22:02:34.703169 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.703098 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba8a2354-02ca-46cf-bea7-d97781a505f5-istiod-ca-cert" pod "ba8a2354-02ca-46cf-bea7-d97781a505f5" (UID: "ba8a2354-02ca-46cf-bea7-d97781a505f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:34.703169 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.703125 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-podinfo\") pod \"ba8a2354-02ca-46cf-bea7-d97781a505f5\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " Oct 30 22:02:34.703292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.703205 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-credential-socket\") pod \"91fba568-9213-4aa9-a093-17df3f0c506e\" (UID: \"91fba568-9213-4aa9-a093-17df3f0c506e\") " Oct 30 22:02:34.703292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.703238 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-workload-certs\") pod \"ba8a2354-02ca-46cf-bea7-d97781a505f5\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " Oct 30 22:02:34.703350 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.703324 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-workload-socket" pod "ba8a2354-02ca-46cf-bea7-d97781a505f5" (UID: "ba8a2354-02ca-46cf-bea7-d97781a505f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:34.703350 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.703328 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-envoy\") pod \"ba8a2354-02ca-46cf-bea7-d97781a505f5\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " Oct 30 22:02:34.703411 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.703401 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba8a2354-02ca-46cf-bea7-d97781a505f5-kube-api-access-8mpnz\" (UniqueName: \"kubernetes.io/projected/ba8a2354-02ca-46cf-bea7-d97781a505f5-kube-api-access-8mpnz\") pod \"ba8a2354-02ca-46cf-bea7-d97781a505f5\" (UID: \"ba8a2354-02ca-46cf-bea7-d97781a505f5\") " Oct 30 22:02:34.703727 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.703697 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-workload-certs" pod "91fba568-9213-4aa9-a093-17df3f0c506e" (UID: "91fba568-9213-4aa9-a093-17df3f0c506e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:34.704009 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.703976 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.704009 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.703998 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.704146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.704013 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnjzk\" (UniqueName: \"kubernetes.io/projected/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-kube-api-access-pnjzk\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.704146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.704026 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8a2354-02ca-46cf-bea7-d97781a505f5-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.704146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.704039 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.704146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.704054 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.704146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.704067 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.704146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.704081 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.704146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.704094 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.704146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.704109 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.704146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.704124 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.704906 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.704165 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91fba568-9213-4aa9-a093-17df3f0c506e-istiod-ca-cert" pod "91fba568-9213-4aa9-a093-17df3f0c506e" (UID: "91fba568-9213-4aa9-a093-17df3f0c506e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:34.705002 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.704972 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-token" pod "ba8a2354-02ca-46cf-bea7-d97781a505f5" (UID: "ba8a2354-02ca-46cf-bea7-d97781a505f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:34.705231 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.705210 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-credential-socket" pod "91fba568-9213-4aa9-a093-17df3f0c506e" (UID: "91fba568-9213-4aa9-a093-17df3f0c506e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:34.705435 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.705412 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-workload-certs" pod "ba8a2354-02ca-46cf-bea7-d97781a505f5" (UID: "ba8a2354-02ca-46cf-bea7-d97781a505f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:34.706100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.706074 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-envoy" pod "ba8a2354-02ca-46cf-bea7-d97781a505f5" (UID: "ba8a2354-02ca-46cf-bea7-d97781a505f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:34.706792 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.706765 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-podinfo" pod "ba8a2354-02ca-46cf-bea7-d97781a505f5" (UID: "ba8a2354-02ca-46cf-bea7-d97781a505f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:34.707088 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.707066 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8a2354-02ca-46cf-bea7-d97781a505f5-kube-api-access-8mpnz" pod "ba8a2354-02ca-46cf-bea7-d97781a505f5" (UID: "ba8a2354-02ca-46cf-bea7-d97781a505f5"). InnerVolumeSpecName "kube-api-access-8mpnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:34.707288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.707264 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fba568-9213-4aa9-a093-17df3f0c506e-kube-api-access-6nmq5" pod "91fba568-9213-4aa9-a093-17df3f0c506e" (UID: "91fba568-9213-4aa9-a093-17df3f0c506e"). InnerVolumeSpecName "kube-api-access-6nmq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:34.707891 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.707868 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-istio-envoy" pod "91fba568-9213-4aa9-a093-17df3f0c506e" (UID: "91fba568-9213-4aa9-a093-17df3f0c506e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:34.707983 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.707884 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fba568-9213-4aa9-a093-17df3f0c506e-istio-token" pod "91fba568-9213-4aa9-a093-17df3f0c506e" (UID: "91fba568-9213-4aa9-a093-17df3f0c506e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:34.708453 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.708430 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91fba568-9213-4aa9-a093-17df3f0c506e-istio-podinfo" pod "91fba568-9213-4aa9-a093-17df3f0c506e" (UID: "91fba568-9213-4aa9-a093-17df3f0c506e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:34.805549 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.805517 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.806030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.805553 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mpnz\" (UniqueName: \"kubernetes.io/projected/ba8a2354-02ca-46cf-bea7-d97781a505f5-kube-api-access-8mpnz\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.806030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.805565 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.806030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.805574 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nmq5\" (UniqueName: \"kubernetes.io/projected/91fba568-9213-4aa9-a093-17df3f0c506e-kube-api-access-6nmq5\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.806030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.805588 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.806030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.805598 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fba568-9213-4aa9-a093-17df3f0c506e-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.806030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.805606 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fba568-9213-4aa9-a093-17df3f0c506e-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.806030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.805636 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fba568-9213-4aa9-a093-17df3f0c506e-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.806030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.805650 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8a2354-02ca-46cf-bea7-d97781a505f5-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.806030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.805658 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fba568-9213-4aa9-a093-17df3f0c506e-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.806030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.805666 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2354-02ca-46cf-bea7-d97781a505f5-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.986215 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.986183 4335 generic.go:356] "Generic (PLEG): container finished" podID="91fba568-9213-4aa9-a093-17df3f0c506e" containerID="3eb1c251b5a4ccbd3adf3a07e097d1a7388b40950d17b3cc5eafdd045a984d88" exitCode=0 Oct 30 22:02:34.986373 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.986247 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" Oct 30 22:02:34.986373 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.986269 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" event={"ID":"91fba568-9213-4aa9-a093-17df3f0c506e","Type":"ContainerDied","Data":"3eb1c251b5a4ccbd3adf3a07e097d1a7388b40950d17b3cc5eafdd045a984d88"} Oct 30 22:02:34.986373 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.986351 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg" event={"ID":"91fba568-9213-4aa9-a093-17df3f0c506e","Type":"ContainerDied","Data":"463f2b0f674a53530989a1d7e44aac1476ad0a76f551b21a63e8a2cf552a5cdc"} Oct 30 22:02:34.986516 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.986379 4335 scope.go:122] "RemoveContainer" containerID="3eb1c251b5a4ccbd3adf3a07e097d1a7388b40950d17b3cc5eafdd045a984d88" Oct 30 22:02:34.987724 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.987705 4335 generic.go:356] "Generic (PLEG): container finished" podID="ba8a2354-02ca-46cf-bea7-d97781a505f5" containerID="f30deb5e197025c9592cdcc8cb863bb149cef7a08645591e2ef2a9d16fce6e14" exitCode=0 Oct 30 22:02:34.987822 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.987777 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" Oct 30 22:02:34.987895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.987779 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" event={"ID":"ba8a2354-02ca-46cf-bea7-d97781a505f5","Type":"ContainerDied","Data":"f30deb5e197025c9592cdcc8cb863bb149cef7a08645591e2ef2a9d16fce6e14"} Oct 30 22:02:34.987946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.987891 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv" event={"ID":"ba8a2354-02ca-46cf-bea7-d97781a505f5","Type":"ContainerDied","Data":"719326c5e859925ef0a76e319e46230123161afff4f7092179c4fb2388b5ffec"} Oct 30 22:02:34.989393 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.989369 4335 generic.go:356] "Generic (PLEG): container finished" podID="d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" containerID="0e5b2586ce5c1bf9d8a0697c2f9419fc88598a997f998c3ea14dfd1383aed9c1" exitCode=0 Oct 30 22:02:34.989509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.989430 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" Oct 30 22:02:34.989509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.989446 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" event={"ID":"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481","Type":"ContainerDied","Data":"0e5b2586ce5c1bf9d8a0697c2f9419fc88598a997f998c3ea14dfd1383aed9c1"} Oct 30 22:02:34.989663 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.989491 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf" event={"ID":"d1f1ef3d-7992-4f3c-8064-3f3c23bb6481","Type":"ContainerDied","Data":"84ab5b6f0086cc2306296d9532ab2d39b4882818b8881d6c27d7041215c5d6d1"} Oct 30 22:02:34.998822 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:34.998802 4335 scope.go:122] "RemoveContainer" containerID="f091890b79771482e31d2f6a353e6e698c91a23f792aae79e8ffcce605ab2441" Oct 30 22:02:35.009006 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:35.008987 4335 scope.go:122] "RemoveContainer" containerID="3eb1c251b5a4ccbd3adf3a07e097d1a7388b40950d17b3cc5eafdd045a984d88" Oct 30 22:02:35.009327 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:02:35.009301 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb1c251b5a4ccbd3adf3a07e097d1a7388b40950d17b3cc5eafdd045a984d88\": container with ID starting with 3eb1c251b5a4ccbd3adf3a07e097d1a7388b40950d17b3cc5eafdd045a984d88 not found: ID does not exist" containerID="3eb1c251b5a4ccbd3adf3a07e097d1a7388b40950d17b3cc5eafdd045a984d88" Oct 30 22:02:35.009401 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:35.009337 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb1c251b5a4ccbd3adf3a07e097d1a7388b40950d17b3cc5eafdd045a984d88"} err="failed to get container status \"3eb1c251b5a4ccbd3adf3a07e097d1a7388b40950d17b3cc5eafdd045a984d88\": rpc error: code = NotFound desc = could not find container \"3eb1c251b5a4ccbd3adf3a07e097d1a7388b40950d17b3cc5eafdd045a984d88\": container with ID starting with 3eb1c251b5a4ccbd3adf3a07e097d1a7388b40950d17b3cc5eafdd045a984d88 not found: ID does not exist" Oct 30 22:02:35.009401 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:35.009361 4335 scope.go:122] "RemoveContainer" containerID="f091890b79771482e31d2f6a353e6e698c91a23f792aae79e8ffcce605ab2441" Oct 30 22:02:35.009684 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:02:35.009663 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f091890b79771482e31d2f6a353e6e698c91a23f792aae79e8ffcce605ab2441\": container with ID starting with f091890b79771482e31d2f6a353e6e698c91a23f792aae79e8ffcce605ab2441 not found: ID does not exist" containerID="f091890b79771482e31d2f6a353e6e698c91a23f792aae79e8ffcce605ab2441" Oct 30 22:02:35.009751 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:35.009692 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f091890b79771482e31d2f6a353e6e698c91a23f792aae79e8ffcce605ab2441"} err="failed to get container status \"f091890b79771482e31d2f6a353e6e698c91a23f792aae79e8ffcce605ab2441\": rpc error: code = NotFound desc = could not find container \"f091890b79771482e31d2f6a353e6e698c91a23f792aae79e8ffcce605ab2441\": container with ID starting with f091890b79771482e31d2f6a353e6e698c91a23f792aae79e8ffcce605ab2441 not found: ID does not exist" Oct 30 22:02:35.009751 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:35.009710 4335 scope.go:122] "RemoveContainer" containerID="f30deb5e197025c9592cdcc8cb863bb149cef7a08645591e2ef2a9d16fce6e14" Oct 30 22:02:35.020818 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:35.020763 4335 scope.go:122] "RemoveContainer" containerID="f30deb5e197025c9592cdcc8cb863bb149cef7a08645591e2ef2a9d16fce6e14" Oct 30 22:02:35.021087 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:02:35.021057 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f30deb5e197025c9592cdcc8cb863bb149cef7a08645591e2ef2a9d16fce6e14\": container with ID starting with f30deb5e197025c9592cdcc8cb863bb149cef7a08645591e2ef2a9d16fce6e14 not found: ID does not exist" containerID="f30deb5e197025c9592cdcc8cb863bb149cef7a08645591e2ef2a9d16fce6e14" Oct 30 22:02:35.021191 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:35.021109 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f30deb5e197025c9592cdcc8cb863bb149cef7a08645591e2ef2a9d16fce6e14"} err="failed to get container status \"f30deb5e197025c9592cdcc8cb863bb149cef7a08645591e2ef2a9d16fce6e14\": rpc error: code = NotFound desc = could not find container \"f30deb5e197025c9592cdcc8cb863bb149cef7a08645591e2ef2a9d16fce6e14\": container with ID starting with f30deb5e197025c9592cdcc8cb863bb149cef7a08645591e2ef2a9d16fce6e14 not found: ID does not exist" Oct 30 22:02:35.021191 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:35.021129 4335 scope.go:122] "RemoveContainer" containerID="0e5b2586ce5c1bf9d8a0697c2f9419fc88598a997f998c3ea14dfd1383aed9c1" Oct 30 22:02:35.033972 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:35.033795 4335 scope.go:122] "RemoveContainer" containerID="0e5b2586ce5c1bf9d8a0697c2f9419fc88598a997f998c3ea14dfd1383aed9c1" Oct 30 22:02:35.035778 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:02:35.035746 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5b2586ce5c1bf9d8a0697c2f9419fc88598a997f998c3ea14dfd1383aed9c1\": container with ID starting with 0e5b2586ce5c1bf9d8a0697c2f9419fc88598a997f998c3ea14dfd1383aed9c1 not found: ID does not exist" containerID="0e5b2586ce5c1bf9d8a0697c2f9419fc88598a997f998c3ea14dfd1383aed9c1" Oct 30 22:02:35.035901 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:35.035782 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5b2586ce5c1bf9d8a0697c2f9419fc88598a997f998c3ea14dfd1383aed9c1"} err="failed to get container status \"0e5b2586ce5c1bf9d8a0697c2f9419fc88598a997f998c3ea14dfd1383aed9c1\": rpc error: code = NotFound desc = could not find container \"0e5b2586ce5c1bf9d8a0697c2f9419fc88598a997f998c3ea14dfd1383aed9c1\": container with ID starting with 0e5b2586ce5c1bf9d8a0697c2f9419fc88598a997f998c3ea14dfd1383aed9c1 not found: ID does not exist" Oct 30 22:02:35.152569 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:35.152507 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg"] Oct 30 22:02:35.158491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:35.158453 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hbnxg"] Oct 30 22:02:35.267092 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:35.267061 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf"] Oct 30 22:02:35.271404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:35.271382 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w8jkf"] Oct 30 22:02:35.380167 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:35.380142 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv"] Oct 30 22:02:35.386357 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:35.386334 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sbvnv"] Oct 30 22:02:37.058072 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:37.058041 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91fba568-9213-4aa9-a093-17df3f0c506e" path="/var/lib/kubelet/pods/91fba568-9213-4aa9-a093-17df3f0c506e/volumes" Oct 30 22:02:37.058484 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:37.058469 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba8a2354-02ca-46cf-bea7-d97781a505f5" path="/var/lib/kubelet/pods/ba8a2354-02ca-46cf-bea7-d97781a505f5/volumes" Oct 30 22:02:37.058897 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:37.058882 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" path="/var/lib/kubelet/pods/d1f1ef3d-7992-4f3c-8064-3f3c23bb6481/volumes" Oct 30 22:02:40.031708 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.031674 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4"] Oct 30 22:02:40.032172 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.032160 4335 cpu_manager.go:414] "containerMap: removing container" podUID="d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" containerName="istio-proxy" Oct 30 22:02:40.032215 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.032173 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" containerName="istio-proxy" Oct 30 22:02:40.032215 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.032194 4335 cpu_manager.go:414] "containerMap: removing container" podUID="91fba568-9213-4aa9-a093-17df3f0c506e" containerName="istio-proxy" Oct 30 22:02:40.032215 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.032200 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91fba568-9213-4aa9-a093-17df3f0c506e" containerName="istio-proxy" Oct 30 22:02:40.032215 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.032210 4335 cpu_manager.go:414] "containerMap: removing container" podUID="91fba568-9213-4aa9-a093-17df3f0c506e" containerName="istio-proxy" Oct 30 22:02:40.032215 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.032216 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91fba568-9213-4aa9-a093-17df3f0c506e" containerName="istio-proxy" Oct 30 22:02:40.032362 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.032230 4335 cpu_manager.go:414] "containerMap: removing container" podUID="ba8a2354-02ca-46cf-bea7-d97781a505f5" containerName="istio-proxy" Oct 30 22:02:40.032362 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.032235 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba8a2354-02ca-46cf-bea7-d97781a505f5" containerName="istio-proxy" Oct 30 22:02:40.032362 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.032315 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1f1ef3d-7992-4f3c-8064-3f3c23bb6481" containerName="istio-proxy" Oct 30 22:02:40.032362 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.032326 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="91fba568-9213-4aa9-a093-17df3f0c506e" containerName="istio-proxy" Oct 30 22:02:40.032362 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.032331 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="91fba568-9213-4aa9-a093-17df3f0c506e" containerName="istio-proxy" Oct 30 22:02:40.032362 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.032338 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba8a2354-02ca-46cf-bea7-d97781a505f5" containerName="istio-proxy" Oct 30 22:02:40.037468 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.037447 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4" Oct 30 22:02:40.094818 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.094791 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4"] Oct 30 22:02:40.138553 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.138515 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:40.138730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.138582 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:02:40.139359 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.139343 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:02:40.139359 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.139343 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:02:40.158868 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.158835 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlx72\" (UniqueName: \"kubernetes.io/projected/00a026eb-0f67-4844-bf37-ca7b42807e0b-kube-api-access-dlx72\") pod \"maas-api-revoke-sub-4c1acdff-prtl4\" (UID: \"00a026eb-0f67-4844-bf37-ca7b42807e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4" Oct 30 22:02:40.159012 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.158994 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a026eb-0f67-4844-bf37-ca7b42807e0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c1acdff-prtl4\" (UID: \"00a026eb-0f67-4844-bf37-ca7b42807e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4" Oct 30 22:02:40.159092 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.159078 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a026eb-0f67-4844-bf37-ca7b42807e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c1acdff-prtl4\" (UID: \"00a026eb-0f67-4844-bf37-ca7b42807e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4" Oct 30 22:02:40.259964 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.259937 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a026eb-0f67-4844-bf37-ca7b42807e0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c1acdff-prtl4\" (UID: \"00a026eb-0f67-4844-bf37-ca7b42807e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4" Oct 30 22:02:40.260093 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.259979 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a026eb-0f67-4844-bf37-ca7b42807e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c1acdff-prtl4\" (UID: \"00a026eb-0f67-4844-bf37-ca7b42807e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4" Oct 30 22:02:40.260093 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.260027 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlx72\" (UniqueName: \"kubernetes.io/projected/00a026eb-0f67-4844-bf37-ca7b42807e0b-kube-api-access-dlx72\") pod \"maas-api-revoke-sub-4c1acdff-prtl4\" (UID: \"00a026eb-0f67-4844-bf37-ca7b42807e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4" Oct 30 22:02:40.260882 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.260866 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a026eb-0f67-4844-bf37-ca7b42807e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c1acdff-prtl4\" (UID: \"00a026eb-0f67-4844-bf37-ca7b42807e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4" Oct 30 22:02:40.262594 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.262574 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a026eb-0f67-4844-bf37-ca7b42807e0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c1acdff-prtl4\" (UID: \"00a026eb-0f67-4844-bf37-ca7b42807e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4" Oct 30 22:02:40.320948 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.320919 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlx72\" (UniqueName: \"kubernetes.io/projected/00a026eb-0f67-4844-bf37-ca7b42807e0b-kube-api-access-dlx72\") pod \"maas-api-revoke-sub-4c1acdff-prtl4\" (UID: \"00a026eb-0f67-4844-bf37-ca7b42807e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4" Oct 30 22:02:40.350582 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.350561 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4" Oct 30 22:02:40.727459 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:40.727429 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4"] Oct 30 22:02:40.728219 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:02:40.728196 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a026eb_0f67_4844_bf37_ca7b42807e0b.slice/crio-aa7e19cf0f05d37f92e3042081d9371aa9f6d5a514059cb64ac76beda69a0e15 WatchSource:0}: Error finding container aa7e19cf0f05d37f92e3042081d9371aa9f6d5a514059cb64ac76beda69a0e15: Status 404 returned error can't find the container with id aa7e19cf0f05d37f92e3042081d9371aa9f6d5a514059cb64ac76beda69a0e15 Oct 30 22:02:41.038320 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:41.038296 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4" event={"ID":"00a026eb-0f67-4844-bf37-ca7b42807e0b","Type":"ContainerStarted","Data":"079cf20760839acc57297b32fce647d3dd31b591b121853ba96969c3563bb9b7"} Oct 30 22:02:41.038664 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:41.038345 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4" event={"ID":"00a026eb-0f67-4844-bf37-ca7b42807e0b","Type":"ContainerStarted","Data":"aa7e19cf0f05d37f92e3042081d9371aa9f6d5a514059cb64ac76beda69a0e15"} Oct 30 22:02:41.155427 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:41.155375 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4" podStartSLOduration=2.155360638 podStartE2EDuration="2.155360638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:41.153203858 +0000 UTC m=+1094.253635591" watchObservedRunningTime="2026-10-30 22:02:41.155360638 +0000 UTC m=+1094.255792371" Oct 30 22:02:42.057294 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:42.057249 4335 generic.go:356] "Generic (PLEG): container finished" podID="00a026eb-0f67-4844-bf37-ca7b42807e0b" containerID="079cf20760839acc57297b32fce647d3dd31b591b121853ba96969c3563bb9b7" exitCode=0 Oct 30 22:02:42.057669 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:42.057327 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4" event={"ID":"00a026eb-0f67-4844-bf37-ca7b42807e0b","Type":"ContainerDied","Data":"079cf20760839acc57297b32fce647d3dd31b591b121853ba96969c3563bb9b7"} Oct 30 22:02:43.216136 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:43.216113 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4" Oct 30 22:02:43.390046 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:43.389961 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a026eb-0f67-4844-bf37-ca7b42807e0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a026eb-0f67-4844-bf37-ca7b42807e0b-maas-api-openshift-ca\") pod \"00a026eb-0f67-4844-bf37-ca7b42807e0b\" (UID: \"00a026eb-0f67-4844-bf37-ca7b42807e0b\") " Oct 30 22:02:43.390208 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:43.390057 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a026eb-0f67-4844-bf37-ca7b42807e0b-kube-api-access-dlx72\" (UniqueName: \"kubernetes.io/projected/00a026eb-0f67-4844-bf37-ca7b42807e0b-kube-api-access-dlx72\") pod \"00a026eb-0f67-4844-bf37-ca7b42807e0b\" (UID: \"00a026eb-0f67-4844-bf37-ca7b42807e0b\") " Oct 30 22:02:43.390208 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:43.390140 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00a026eb-0f67-4844-bf37-ca7b42807e0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a026eb-0f67-4844-bf37-ca7b42807e0b-maas-api-xks-ca\") pod \"00a026eb-0f67-4844-bf37-ca7b42807e0b\" (UID: \"00a026eb-0f67-4844-bf37-ca7b42807e0b\") " Oct 30 22:02:43.390309 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:43.390282 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a026eb-0f67-4844-bf37-ca7b42807e0b-maas-api-openshift-ca" pod "00a026eb-0f67-4844-bf37-ca7b42807e0b" (UID: "00a026eb-0f67-4844-bf37-ca7b42807e0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:43.390521 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:43.390505 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a026eb-0f67-4844-bf37-ca7b42807e0b-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:43.392275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:43.392254 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a026eb-0f67-4844-bf37-ca7b42807e0b-kube-api-access-dlx72" pod "00a026eb-0f67-4844-bf37-ca7b42807e0b" (UID: "00a026eb-0f67-4844-bf37-ca7b42807e0b"). InnerVolumeSpecName "kube-api-access-dlx72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:43.392275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:43.392266 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a026eb-0f67-4844-bf37-ca7b42807e0b-maas-api-xks-ca" pod "00a026eb-0f67-4844-bf37-ca7b42807e0b" (UID: "00a026eb-0f67-4844-bf37-ca7b42807e0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:43.491816 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:43.491783 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlx72\" (UniqueName: \"kubernetes.io/projected/00a026eb-0f67-4844-bf37-ca7b42807e0b-kube-api-access-dlx72\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:43.491816 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:43.491813 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a026eb-0f67-4844-bf37-ca7b42807e0b-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:44.073218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:44.073183 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4" event={"ID":"00a026eb-0f67-4844-bf37-ca7b42807e0b","Type":"ContainerDied","Data":"aa7e19cf0f05d37f92e3042081d9371aa9f6d5a514059cb64ac76beda69a0e15"} Oct 30 22:02:44.073361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:44.073222 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4" Oct 30 22:02:44.073361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:44.073238 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa7e19cf0f05d37f92e3042081d9371aa9f6d5a514059cb64ac76beda69a0e15" Oct 30 22:02:53.531464 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:53.531433 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:02:55.321298 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:02:55.321258 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:02:55.779679 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.779591 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf"] Oct 30 22:02:55.780215 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.780196 4335 cpu_manager.go:414] "containerMap: removing container" podUID="00a026eb-0f67-4844-bf37-ca7b42807e0b" containerName="revoke-keys" Oct 30 22:02:55.780297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.780217 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a026eb-0f67-4844-bf37-ca7b42807e0b" containerName="revoke-keys" Oct 30 22:02:55.780336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.780325 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a026eb-0f67-4844-bf37-ca7b42807e0b" containerName="revoke-keys" Oct 30 22:02:55.785065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.785044 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf" Oct 30 22:02:55.811291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.811268 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08eb5248-5ccb-4202-baed-c2c0651f3cfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b06dfc7d-lqgsf\" (UID: \"08eb5248-5ccb-4202-baed-c2c0651f3cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf" Oct 30 22:02:55.811391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.811310 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08eb5248-5ccb-4202-baed-c2c0651f3cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b06dfc7d-lqgsf\" (UID: \"08eb5248-5ccb-4202-baed-c2c0651f3cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf" Oct 30 22:02:55.811448 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.811427 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdsrv\" (UniqueName: \"kubernetes.io/projected/08eb5248-5ccb-4202-baed-c2c0651f3cfb-kube-api-access-cdsrv\") pod \"maas-api-revoke-sub-b06dfc7d-lqgsf\" (UID: \"08eb5248-5ccb-4202-baed-c2c0651f3cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf" Oct 30 22:02:55.839456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.839434 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:55.839756 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.839738 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:02:55.839826 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.839812 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:02:55.839876 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.839836 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:02:55.892674 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.892650 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf"] Oct 30 22:02:55.912285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.912261 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08eb5248-5ccb-4202-baed-c2c0651f3cfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b06dfc7d-lqgsf\" (UID: \"08eb5248-5ccb-4202-baed-c2c0651f3cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf" Oct 30 22:02:55.912420 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.912306 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08eb5248-5ccb-4202-baed-c2c0651f3cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b06dfc7d-lqgsf\" (UID: \"08eb5248-5ccb-4202-baed-c2c0651f3cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf" Oct 30 22:02:55.912420 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.912381 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdsrv\" (UniqueName: \"kubernetes.io/projected/08eb5248-5ccb-4202-baed-c2c0651f3cfb-kube-api-access-cdsrv\") pod \"maas-api-revoke-sub-b06dfc7d-lqgsf\" (UID: \"08eb5248-5ccb-4202-baed-c2c0651f3cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf" Oct 30 22:02:55.913068 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.913046 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08eb5248-5ccb-4202-baed-c2c0651f3cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b06dfc7d-lqgsf\" (UID: \"08eb5248-5ccb-4202-baed-c2c0651f3cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf" Oct 30 22:02:55.914981 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.914955 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08eb5248-5ccb-4202-baed-c2c0651f3cfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b06dfc7d-lqgsf\" (UID: \"08eb5248-5ccb-4202-baed-c2c0651f3cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf" Oct 30 22:02:55.973223 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:55.973198 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdsrv\" (UniqueName: \"kubernetes.io/projected/08eb5248-5ccb-4202-baed-c2c0651f3cfb-kube-api-access-cdsrv\") pod \"maas-api-revoke-sub-b06dfc7d-lqgsf\" (UID: \"08eb5248-5ccb-4202-baed-c2c0651f3cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf" Oct 30 22:02:56.098913 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:56.098891 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf" Oct 30 22:02:56.482968 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:56.482941 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf"] Oct 30 22:02:56.483675 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:02:56.483646 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08eb5248_5ccb_4202_baed_c2c0651f3cfb.slice/crio-5fea48e29a53f0c812505dfa6321a3a9027ba33362eb056789b6e0d9c220a955 WatchSource:0}: Error finding container 5fea48e29a53f0c812505dfa6321a3a9027ba33362eb056789b6e0d9c220a955: Status 404 returned error can't find the container with id 5fea48e29a53f0c812505dfa6321a3a9027ba33362eb056789b6e0d9c220a955 Oct 30 22:02:57.173080 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:57.172995 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf" event={"ID":"08eb5248-5ccb-4202-baed-c2c0651f3cfb","Type":"ContainerStarted","Data":"5920b69bc5bd3646d86cb53ecc511d8546acc238fd0537c25f4d0b03fd9b47a3"} Oct 30 22:02:57.173080 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:57.173058 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf" event={"ID":"08eb5248-5ccb-4202-baed-c2c0651f3cfb","Type":"ContainerStarted","Data":"5fea48e29a53f0c812505dfa6321a3a9027ba33362eb056789b6e0d9c220a955"} Oct 30 22:02:57.291268 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:57.291221 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf" podStartSLOduration=2.291207227 podStartE2EDuration="2.291207227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:57.287777702 +0000 UTC m=+1110.388209432" watchObservedRunningTime="2026-10-30 22:02:57.291207227 +0000 UTC m=+1110.391638959" Oct 30 22:02:57.516021 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:57.515954 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq"] Oct 30 22:02:57.520037 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:57.520017 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq" Oct 30 22:02:57.583744 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:57.583717 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq"] Oct 30 22:02:57.631560 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:57.631534 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b43447-cccf-4ecc-9535-26e54a96c27b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd935447-4knkq\" (UID: \"93b43447-cccf-4ecc-9535-26e54a96c27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq" Oct 30 22:02:57.631725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:57.631694 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b43447-cccf-4ecc-9535-26e54a96c27b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd935447-4knkq\" (UID: \"93b43447-cccf-4ecc-9535-26e54a96c27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq" Oct 30 22:02:57.631805 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:57.631734 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jfr9\" (UniqueName: \"kubernetes.io/projected/93b43447-cccf-4ecc-9535-26e54a96c27b-kube-api-access-8jfr9\") pod \"maas-api-revoke-sub-cd935447-4knkq\" (UID: \"93b43447-cccf-4ecc-9535-26e54a96c27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq" Oct 30 22:02:57.732353 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:57.732323 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b43447-cccf-4ecc-9535-26e54a96c27b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd935447-4knkq\" (UID: \"93b43447-cccf-4ecc-9535-26e54a96c27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq" Oct 30 22:02:57.732518 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:57.732417 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b43447-cccf-4ecc-9535-26e54a96c27b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd935447-4knkq\" (UID: \"93b43447-cccf-4ecc-9535-26e54a96c27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq" Oct 30 22:02:57.732518 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:57.732446 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jfr9\" (UniqueName: \"kubernetes.io/projected/93b43447-cccf-4ecc-9535-26e54a96c27b-kube-api-access-8jfr9\") pod \"maas-api-revoke-sub-cd935447-4knkq\" (UID: \"93b43447-cccf-4ecc-9535-26e54a96c27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq" Oct 30 22:02:57.733061 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:57.733035 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b43447-cccf-4ecc-9535-26e54a96c27b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd935447-4knkq\" (UID: \"93b43447-cccf-4ecc-9535-26e54a96c27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq" Oct 30 22:02:57.734999 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:57.734975 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b43447-cccf-4ecc-9535-26e54a96c27b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd935447-4knkq\" (UID: \"93b43447-cccf-4ecc-9535-26e54a96c27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq" Oct 30 22:02:57.792866 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:57.792814 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jfr9\" (UniqueName: \"kubernetes.io/projected/93b43447-cccf-4ecc-9535-26e54a96c27b-kube-api-access-8jfr9\") pod \"maas-api-revoke-sub-cd935447-4knkq\" (UID: \"93b43447-cccf-4ecc-9535-26e54a96c27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq" Oct 30 22:02:57.833121 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:57.833091 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq" Oct 30 22:02:58.180924 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.180889 4335 generic.go:356] "Generic (PLEG): container finished" podID="08eb5248-5ccb-4202-baed-c2c0651f3cfb" containerID="5920b69bc5bd3646d86cb53ecc511d8546acc238fd0537c25f4d0b03fd9b47a3" exitCode=0 Oct 30 22:02:58.181106 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.180926 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf" event={"ID":"08eb5248-5ccb-4202-baed-c2c0651f3cfb","Type":"ContainerDied","Data":"5920b69bc5bd3646d86cb53ecc511d8546acc238fd0537c25f4d0b03fd9b47a3"} Oct 30 22:02:58.218370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.218348 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq"] Oct 30 22:02:58.220017 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:02:58.219984 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b43447_cccf_4ecc_9535_26e54a96c27b.slice/crio-79d3f858337bf06012ee0d3a4e922883f3739333bfc76aadc57dd9e2e7f96255 WatchSource:0}: Error finding container 79d3f858337bf06012ee0d3a4e922883f3739333bfc76aadc57dd9e2e7f96255: Status 404 returned error can't find the container with id 79d3f858337bf06012ee0d3a4e922883f3739333bfc76aadc57dd9e2e7f96255 Oct 30 22:02:58.489422 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.489346 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm"] Oct 30 22:02:58.494575 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.494554 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.541462 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.541432 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a334a2d7-0b91-402c-b690-fff3f5045bfc-istiod-ca-cert\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.541843 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.541471 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-envoy\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.541843 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.541628 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-token\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.541843 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.541749 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-podinfo\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.541843 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.541788 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks6hg\" (UniqueName: \"kubernetes.io/projected/a334a2d7-0b91-402c-b690-fff3f5045bfc-kube-api-access-ks6hg\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.541996 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.541888 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-workload-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.541996 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.541923 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-data\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.542075 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.541993 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-workload-certs\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.542075 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.542029 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-credential-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.565934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.565901 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm"] Oct 30 22:02:58.643606 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.643575 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-token\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.643771 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.643707 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-podinfo\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.643771 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.643737 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks6hg\" (UniqueName: \"kubernetes.io/projected/a334a2d7-0b91-402c-b690-fff3f5045bfc-kube-api-access-ks6hg\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.643850 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.643774 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-workload-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.643850 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.643792 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-data\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.643850 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.643816 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-workload-certs\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.643850 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.643834 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-credential-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.644046 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.643863 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a334a2d7-0b91-402c-b690-fff3f5045bfc-istiod-ca-cert\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.644046 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.643895 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-envoy\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.644279 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.644255 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-credential-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.644456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.644433 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-workload-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.644552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.644507 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-workload-certs\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.644780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.644762 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-data\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.644867 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.644855 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a334a2d7-0b91-402c-b690-fff3f5045bfc-istiod-ca-cert\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.646895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.646868 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-envoy\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.647088 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.647071 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-podinfo\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.704691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.704667 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-token\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.704872 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.704854 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks6hg\" (UniqueName: \"kubernetes.io/projected/a334a2d7-0b91-402c-b690-fff3f5045bfc-kube-api-access-ks6hg\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:58.812714 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:58.812679 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:02:59.189143 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:59.189113 4335 generic.go:356] "Generic (PLEG): container finished" podID="93b43447-cccf-4ecc-9535-26e54a96c27b" containerID="9eaffe32f204bdf513ef221383801424158b43b30e656fd9cc5fee58c5da46aa" exitCode=0 Oct 30 22:02:59.189259 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:59.189197 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq" event={"ID":"93b43447-cccf-4ecc-9535-26e54a96c27b","Type":"ContainerDied","Data":"9eaffe32f204bdf513ef221383801424158b43b30e656fd9cc5fee58c5da46aa"} Oct 30 22:02:59.189259 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:59.189256 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq" event={"ID":"93b43447-cccf-4ecc-9535-26e54a96c27b","Type":"ContainerStarted","Data":"79d3f858337bf06012ee0d3a4e922883f3739333bfc76aadc57dd9e2e7f96255"} Oct 30 22:02:59.244736 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:59.244702 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm"] Oct 30 22:02:59.254838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:59.254804 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:59.254942 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:59.254887 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:59.254942 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:59.254934 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:59.345847 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:59.345824 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf" Oct 30 22:02:59.455130 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:59.455052 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08eb5248-5ccb-4202-baed-c2c0651f3cfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08eb5248-5ccb-4202-baed-c2c0651f3cfb-maas-api-xks-ca\") pod \"08eb5248-5ccb-4202-baed-c2c0651f3cfb\" (UID: \"08eb5248-5ccb-4202-baed-c2c0651f3cfb\") " Oct 30 22:02:59.455273 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:59.455195 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08eb5248-5ccb-4202-baed-c2c0651f3cfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08eb5248-5ccb-4202-baed-c2c0651f3cfb-maas-api-openshift-ca\") pod \"08eb5248-5ccb-4202-baed-c2c0651f3cfb\" (UID: \"08eb5248-5ccb-4202-baed-c2c0651f3cfb\") " Oct 30 22:02:59.455273 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:59.455229 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08eb5248-5ccb-4202-baed-c2c0651f3cfb-kube-api-access-cdsrv\" (UniqueName: \"kubernetes.io/projected/08eb5248-5ccb-4202-baed-c2c0651f3cfb-kube-api-access-cdsrv\") pod \"08eb5248-5ccb-4202-baed-c2c0651f3cfb\" (UID: \"08eb5248-5ccb-4202-baed-c2c0651f3cfb\") " Oct 30 22:02:59.455559 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:59.455536 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08eb5248-5ccb-4202-baed-c2c0651f3cfb-maas-api-openshift-ca" pod "08eb5248-5ccb-4202-baed-c2c0651f3cfb" (UID: "08eb5248-5ccb-4202-baed-c2c0651f3cfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:59.455760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:59.455740 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08eb5248-5ccb-4202-baed-c2c0651f3cfb-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:59.457388 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:59.457360 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08eb5248-5ccb-4202-baed-c2c0651f3cfb-maas-api-xks-ca" pod "08eb5248-5ccb-4202-baed-c2c0651f3cfb" (UID: "08eb5248-5ccb-4202-baed-c2c0651f3cfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:59.457574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:59.457551 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08eb5248-5ccb-4202-baed-c2c0651f3cfb-kube-api-access-cdsrv" pod "08eb5248-5ccb-4202-baed-c2c0651f3cfb" (UID: "08eb5248-5ccb-4202-baed-c2c0651f3cfb"). InnerVolumeSpecName "kube-api-access-cdsrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:59.557379 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:59.557345 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08eb5248-5ccb-4202-baed-c2c0651f3cfb-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:59.557379 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:02:59.557371 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdsrv\" (UniqueName: \"kubernetes.io/projected/08eb5248-5ccb-4202-baed-c2c0651f3cfb-kube-api-access-cdsrv\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:00.220707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.217365 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf" event={"ID":"08eb5248-5ccb-4202-baed-c2c0651f3cfb","Type":"ContainerDied","Data":"5fea48e29a53f0c812505dfa6321a3a9027ba33362eb056789b6e0d9c220a955"} Oct 30 22:03:00.220707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.217427 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fea48e29a53f0c812505dfa6321a3a9027ba33362eb056789b6e0d9c220a955" Oct 30 22:03:00.220707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.217492 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf" Oct 30 22:03:00.227029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.226997 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" event={"ID":"a334a2d7-0b91-402c-b690-fff3f5045bfc","Type":"ContainerStarted","Data":"a30754a4b327226fd220b04950b5f442057d824728224734be3b091487ff6722"} Oct 30 22:03:00.227170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.227057 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" event={"ID":"a334a2d7-0b91-402c-b690-fff3f5045bfc","Type":"ContainerStarted","Data":"c34149db5347c66d61c16a18530c3415ddb20762e5953f53b44ee2033dcfca67"} Oct 30 22:03:00.388452 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.388430 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq" Oct 30 22:03:00.424498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.424431 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" podStartSLOduration=2.424410403 podStartE2EDuration="2.424410403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:00.415125029 +0000 UTC m=+1113.515556762" watchObservedRunningTime="2026-10-30 22:03:00.424410403 +0000 UTC m=+1113.524842139" Oct 30 22:03:00.469262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.469237 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93b43447-cccf-4ecc-9535-26e54a96c27b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b43447-cccf-4ecc-9535-26e54a96c27b-maas-api-xks-ca\") pod \"93b43447-cccf-4ecc-9535-26e54a96c27b\" (UID: \"93b43447-cccf-4ecc-9535-26e54a96c27b\") " Oct 30 22:03:00.469412 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.469331 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b43447-cccf-4ecc-9535-26e54a96c27b-kube-api-access-8jfr9\" (UniqueName: \"kubernetes.io/projected/93b43447-cccf-4ecc-9535-26e54a96c27b-kube-api-access-8jfr9\") pod \"93b43447-cccf-4ecc-9535-26e54a96c27b\" (UID: \"93b43447-cccf-4ecc-9535-26e54a96c27b\") " Oct 30 22:03:00.469412 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.469383 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93b43447-cccf-4ecc-9535-26e54a96c27b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b43447-cccf-4ecc-9535-26e54a96c27b-maas-api-openshift-ca\") pod \"93b43447-cccf-4ecc-9535-26e54a96c27b\" (UID: \"93b43447-cccf-4ecc-9535-26e54a96c27b\") " Oct 30 22:03:00.469794 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.469773 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b43447-cccf-4ecc-9535-26e54a96c27b-maas-api-openshift-ca" pod "93b43447-cccf-4ecc-9535-26e54a96c27b" (UID: "93b43447-cccf-4ecc-9535-26e54a96c27b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:00.471562 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.471515 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b43447-cccf-4ecc-9535-26e54a96c27b-kube-api-access-8jfr9" pod "93b43447-cccf-4ecc-9535-26e54a96c27b" (UID: "93b43447-cccf-4ecc-9535-26e54a96c27b"). InnerVolumeSpecName "kube-api-access-8jfr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:00.471562 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.471533 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93b43447-cccf-4ecc-9535-26e54a96c27b-maas-api-xks-ca" pod "93b43447-cccf-4ecc-9535-26e54a96c27b" (UID: "93b43447-cccf-4ecc-9535-26e54a96c27b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:00.570718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.570686 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jfr9\" (UniqueName: \"kubernetes.io/projected/93b43447-cccf-4ecc-9535-26e54a96c27b-kube-api-access-8jfr9\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:00.570718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.570712 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b43447-cccf-4ecc-9535-26e54a96c27b-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:00.570718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.570723 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b43447-cccf-4ecc-9535-26e54a96c27b-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:00.820797 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.820764 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:03:00.822803 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.822774 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:03:00.822931 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:00.822823 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" podUID="a334a2d7-0b91-402c-b690-fff3f5045bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 22:03:01.099683 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:03:01.095863 4335 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b43447_cccf_4ecc_9535_26e54a96c27b.slice\": RecentStats: unable to find data in memory cache]" Oct 30 22:03:01.236891 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:01.236867 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq" Oct 30 22:03:01.237053 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:01.236906 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq" event={"ID":"93b43447-cccf-4ecc-9535-26e54a96c27b","Type":"ContainerDied","Data":"79d3f858337bf06012ee0d3a4e922883f3739333bfc76aadc57dd9e2e7f96255"} Oct 30 22:03:01.237053 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:01.236959 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79d3f858337bf06012ee0d3a4e922883f3739333bfc76aadc57dd9e2e7f96255" Oct 30 22:03:01.815997 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:01.815960 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:03:01.816588 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:01.816554 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" podUID="a334a2d7-0b91-402c-b690-fff3f5045bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 22:03:02.814638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:02.813937 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:03:02.814638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:02.814008 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" podUID="a334a2d7-0b91-402c-b690-fff3f5045bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 22:03:03.813954 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:03.813918 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:03:03.814582 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:03.814549 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" podUID="a334a2d7-0b91-402c-b690-fff3f5045bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 22:03:04.821024 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:04.820984 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:03:04.821522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:04.821041 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" podUID="a334a2d7-0b91-402c-b690-fff3f5045bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 22:03:05.885376 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:05.884297 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:03:05.885967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:05.885530 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:03:05.886722 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:05.886699 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:03:13.887340 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:13.887111 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv"] Oct 30 22:03:13.889079 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:13.889058 4335 cpu_manager.go:414] "containerMap: removing container" podUID="93b43447-cccf-4ecc-9535-26e54a96c27b" containerName="revoke-keys" Oct 30 22:03:13.889290 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:13.889255 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b43447-cccf-4ecc-9535-26e54a96c27b" containerName="revoke-keys" Oct 30 22:03:13.889492 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:13.889480 4335 cpu_manager.go:414] "containerMap: removing container" podUID="08eb5248-5ccb-4202-baed-c2c0651f3cfb" containerName="revoke-keys" Oct 30 22:03:13.889642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:13.889630 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08eb5248-5ccb-4202-baed-c2c0651f3cfb" containerName="revoke-keys" Oct 30 22:03:13.895069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:13.895047 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b43447-cccf-4ecc-9535-26e54a96c27b" containerName="revoke-keys" Oct 30 22:03:13.895214 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:13.895204 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="08eb5248-5ccb-4202-baed-c2c0651f3cfb" containerName="revoke-keys" Oct 30 22:03:13.899299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:13.899275 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:13.957536 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:13.955633 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7df73b-openshift-default-dockercfg-r7qb2\"" Oct 30 22:03:14.011605 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.011563 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv"] Oct 30 22:03:14.038074 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.037601 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-data\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.038074 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.037701 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-workload-socket\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.038074 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.037751 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-podinfo\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.038074 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.037817 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-envoy\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.038074 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.037856 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-credential-socket\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.038074 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.037889 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2f07ba-2e3c-45b2-b619-7ce607083590-istiod-ca-cert\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.038074 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.037930 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-token\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.038074 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.038010 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-workload-certs\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.038074 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.038044 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqfmf\" (UniqueName: \"kubernetes.io/projected/6c2f07ba-2e3c-45b2-b619-7ce607083590-kube-api-access-wqfmf\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.152707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.152266 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-workload-certs\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.152707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.152329 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqfmf\" (UniqueName: \"kubernetes.io/projected/6c2f07ba-2e3c-45b2-b619-7ce607083590-kube-api-access-wqfmf\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.152707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.152402 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-data\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.152707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.152455 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-workload-socket\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.152707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.152503 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-podinfo\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.152707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.152578 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-envoy\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.152707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.152636 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-credential-socket\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.152707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.152671 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2f07ba-2e3c-45b2-b619-7ce607083590-istiod-ca-cert\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.153254 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.152719 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-token\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.156637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.153484 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-workload-certs\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.156637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.154003 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-data\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.156637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.154234 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-workload-socket\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.167190 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.167166 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-credential-socket\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.177115 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.176791 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-podinfo\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.178853 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.178824 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2f07ba-2e3c-45b2-b619-7ce607083590-istiod-ca-cert\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.186882 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.184823 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-envoy\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.226641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.224347 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqfmf\" (UniqueName: \"kubernetes.io/projected/6c2f07ba-2e3c-45b2-b619-7ce607083590-kube-api-access-wqfmf\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.228829 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.228769 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-token\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.524132 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.524042 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:14.986865 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:03:14.986830 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2f07ba_2e3c_45b2_b619_7ce607083590.slice/crio-3873de063c980302910ad46cfadd65aac7daedb71d55fcadb1c93b3b3eac47ac WatchSource:0}: Error finding container 3873de063c980302910ad46cfadd65aac7daedb71d55fcadb1c93b3b3eac47ac: Status 404 returned error can't find the container with id 3873de063c980302910ad46cfadd65aac7daedb71d55fcadb1c93b3b3eac47ac Oct 30 22:03:14.986865 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.986854 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv"] Oct 30 22:03:14.989201 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.989174 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:03:14.989301 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.989230 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:03:14.989301 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:14.989260 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:03:15.347419 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:15.347385 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" event={"ID":"6c2f07ba-2e3c-45b2-b619-7ce607083590","Type":"ContainerStarted","Data":"c18d86f1d774c013b51aa4ce8f4b3490e0029ebabc8e430e29da1f04aa3fdaeb"} Oct 30 22:03:15.347585 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:15.347457 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" event={"ID":"6c2f07ba-2e3c-45b2-b619-7ce607083590","Type":"ContainerStarted","Data":"3873de063c980302910ad46cfadd65aac7daedb71d55fcadb1c93b3b3eac47ac"} Oct 30 22:03:15.469479 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:15.469422 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" podStartSLOduration=2.469404634 podStartE2EDuration="2.469404634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:15.468044716 +0000 UTC m=+1128.568476448" watchObservedRunningTime="2026-10-30 22:03:15.469404634 +0000 UTC m=+1128.569836367" Oct 30 22:03:15.525643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:15.525143 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:16.526501 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:16.526469 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 22:03:16.526895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:16.526525 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" podUID="6c2f07ba-2e3c-45b2-b619-7ce607083590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 22:03:17.525035 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:17.525002 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 22:03:17.525209 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:17.525055 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" podUID="6c2f07ba-2e3c-45b2-b619-7ce607083590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 22:03:18.525052 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:18.525012 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 22:03:18.525573 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:18.525075 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" podUID="6c2f07ba-2e3c-45b2-b619-7ce607083590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 22:03:19.525017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:19.524980 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 22:03:19.525197 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:19.525040 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" podUID="6c2f07ba-2e3c-45b2-b619-7ce607083590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 22:03:20.525033 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:20.524998 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 22:03:20.525205 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:20.525052 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" podUID="6c2f07ba-2e3c-45b2-b619-7ce607083590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 22:03:21.568052 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:21.567972 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:21.568520 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:21.568386 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:21.569130 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:21.569108 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:03:23.311564 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:23.311527 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs"] Oct 30 22:03:23.316331 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:23.316299 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs" Oct 30 22:03:23.370764 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:23.370742 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs"] Oct 30 22:03:23.415443 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:23.415417 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:23.415443 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:23.415430 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:23.415443 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:23.415430 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:23.417165 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:23.417148 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:03:23.470159 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:23.470136 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4p8h\" (UniqueName: \"kubernetes.io/projected/d1708729-a250-4d83-b4b8-1988c3630a58-kube-api-access-c4p8h\") pod \"maas-api-revoke-sub-e683f4f1-hkfqs\" (UID: \"d1708729-a250-4d83-b4b8-1988c3630a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs" Oct 30 22:03:23.470278 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:23.470186 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1708729-a250-4d83-b4b8-1988c3630a58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e683f4f1-hkfqs\" (UID: \"d1708729-a250-4d83-b4b8-1988c3630a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs" Oct 30 22:03:23.470333 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:23.470316 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1708729-a250-4d83-b4b8-1988c3630a58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e683f4f1-hkfqs\" (UID: \"d1708729-a250-4d83-b4b8-1988c3630a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs" Oct 30 22:03:23.571476 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:23.571403 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4p8h\" (UniqueName: \"kubernetes.io/projected/d1708729-a250-4d83-b4b8-1988c3630a58-kube-api-access-c4p8h\") pod \"maas-api-revoke-sub-e683f4f1-hkfqs\" (UID: \"d1708729-a250-4d83-b4b8-1988c3630a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs" Oct 30 22:03:23.571476 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:23.571455 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1708729-a250-4d83-b4b8-1988c3630a58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e683f4f1-hkfqs\" (UID: \"d1708729-a250-4d83-b4b8-1988c3630a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs" Oct 30 22:03:23.571735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:23.571506 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1708729-a250-4d83-b4b8-1988c3630a58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e683f4f1-hkfqs\" (UID: \"d1708729-a250-4d83-b4b8-1988c3630a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs" Oct 30 22:03:23.572254 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:23.572230 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1708729-a250-4d83-b4b8-1988c3630a58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e683f4f1-hkfqs\" (UID: \"d1708729-a250-4d83-b4b8-1988c3630a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs" Oct 30 22:03:23.574065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:23.574046 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1708729-a250-4d83-b4b8-1988c3630a58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e683f4f1-hkfqs\" (UID: \"d1708729-a250-4d83-b4b8-1988c3630a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs" Oct 30 22:03:23.632601 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:23.632552 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4p8h\" (UniqueName: \"kubernetes.io/projected/d1708729-a250-4d83-b4b8-1988c3630a58-kube-api-access-c4p8h\") pod \"maas-api-revoke-sub-e683f4f1-hkfqs\" (UID: \"d1708729-a250-4d83-b4b8-1988c3630a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs" Oct 30 22:03:23.632777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:23.632744 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs" Oct 30 22:03:24.014297 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:03:24.014255 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1708729_a250_4d83_b4b8_1988c3630a58.slice/crio-014a5d9eac2cb56f9535d30d10fd50c79baf794f2dc0f982d77d50ca9e9fb531 WatchSource:0}: Error finding container 014a5d9eac2cb56f9535d30d10fd50c79baf794f2dc0f982d77d50ca9e9fb531: Status 404 returned error can't find the container with id 014a5d9eac2cb56f9535d30d10fd50c79baf794f2dc0f982d77d50ca9e9fb531 Oct 30 22:03:24.015560 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:24.015534 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs"] Oct 30 22:03:24.418793 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:24.418758 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs" event={"ID":"d1708729-a250-4d83-b4b8-1988c3630a58","Type":"ContainerStarted","Data":"21860c68681be48163131b9891364893a73305ac5c6b7d1041fe68e9cce73b57"} Oct 30 22:03:24.419198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:24.418824 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs" event={"ID":"d1708729-a250-4d83-b4b8-1988c3630a58","Type":"ContainerStarted","Data":"014a5d9eac2cb56f9535d30d10fd50c79baf794f2dc0f982d77d50ca9e9fb531"} Oct 30 22:03:24.535262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:24.535210 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs" podStartSLOduration=1.5351963 podStartE2EDuration="1.5351963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:24.533077219 +0000 UTC m=+1137.633508952" watchObservedRunningTime="2026-10-30 22:03:24.5351963 +0000 UTC m=+1137.635628032" Oct 30 22:03:25.322878 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:03:25.322830 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:03:25.428843 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:25.428800 4335 generic.go:356] "Generic (PLEG): container finished" podID="d1708729-a250-4d83-b4b8-1988c3630a58" containerID="21860c68681be48163131b9891364893a73305ac5c6b7d1041fe68e9cce73b57" exitCode=0 Oct 30 22:03:25.429235 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:25.428942 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs" event={"ID":"d1708729-a250-4d83-b4b8-1988c3630a58","Type":"ContainerDied","Data":"21860c68681be48163131b9891364893a73305ac5c6b7d1041fe68e9cce73b57"} Oct 30 22:03:26.581862 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:26.581841 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs" Oct 30 22:03:26.602044 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:26.602016 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1708729-a250-4d83-b4b8-1988c3630a58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1708729-a250-4d83-b4b8-1988c3630a58-maas-api-xks-ca\") pod \"d1708729-a250-4d83-b4b8-1988c3630a58\" (UID: \"d1708729-a250-4d83-b4b8-1988c3630a58\") " Oct 30 22:03:26.602194 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:26.602048 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1708729-a250-4d83-b4b8-1988c3630a58-kube-api-access-c4p8h\" (UniqueName: \"kubernetes.io/projected/d1708729-a250-4d83-b4b8-1988c3630a58-kube-api-access-c4p8h\") pod \"d1708729-a250-4d83-b4b8-1988c3630a58\" (UID: \"d1708729-a250-4d83-b4b8-1988c3630a58\") " Oct 30 22:03:26.602194 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:26.602084 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1708729-a250-4d83-b4b8-1988c3630a58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1708729-a250-4d83-b4b8-1988c3630a58-maas-api-openshift-ca\") pod \"d1708729-a250-4d83-b4b8-1988c3630a58\" (UID: \"d1708729-a250-4d83-b4b8-1988c3630a58\") " Oct 30 22:03:26.602480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:26.602456 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1708729-a250-4d83-b4b8-1988c3630a58-maas-api-openshift-ca" pod "d1708729-a250-4d83-b4b8-1988c3630a58" (UID: "d1708729-a250-4d83-b4b8-1988c3630a58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:26.604216 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:26.604183 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1708729-a250-4d83-b4b8-1988c3630a58-kube-api-access-c4p8h" pod "d1708729-a250-4d83-b4b8-1988c3630a58" (UID: "d1708729-a250-4d83-b4b8-1988c3630a58"). InnerVolumeSpecName "kube-api-access-c4p8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:26.604216 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:26.604197 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1708729-a250-4d83-b4b8-1988c3630a58-maas-api-xks-ca" pod "d1708729-a250-4d83-b4b8-1988c3630a58" (UID: "d1708729-a250-4d83-b4b8-1988c3630a58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:26.703229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:26.703160 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1708729-a250-4d83-b4b8-1988c3630a58-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:26.703229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:26.703194 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4p8h\" (UniqueName: \"kubernetes.io/projected/d1708729-a250-4d83-b4b8-1988c3630a58-kube-api-access-c4p8h\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:26.703229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:26.703210 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1708729-a250-4d83-b4b8-1988c3630a58-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:27.445900 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:27.445836 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs" Oct 30 22:03:27.445900 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:27.445866 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs" event={"ID":"d1708729-a250-4d83-b4b8-1988c3630a58","Type":"ContainerDied","Data":"014a5d9eac2cb56f9535d30d10fd50c79baf794f2dc0f982d77d50ca9e9fb531"} Oct 30 22:03:27.446045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:27.445915 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="014a5d9eac2cb56f9535d30d10fd50c79baf794f2dc0f982d77d50ca9e9fb531" Oct 30 22:03:28.172303 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.172269 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk"] Oct 30 22:03:28.173188 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.173166 4335 cpu_manager.go:414] "containerMap: removing container" podUID="d1708729-a250-4d83-b4b8-1988c3630a58" containerName="revoke-keys" Oct 30 22:03:28.173256 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.173203 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1708729-a250-4d83-b4b8-1988c3630a58" containerName="revoke-keys" Oct 30 22:03:28.173395 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.173383 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1708729-a250-4d83-b4b8-1988c3630a58" containerName="revoke-keys" Oct 30 22:03:28.178507 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.178486 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk" Oct 30 22:03:28.217943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.217916 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l59dc\" (UniqueName: \"kubernetes.io/projected/f7115d25-0f2d-4679-958b-d31e313b197f-kube-api-access-l59dc\") pod \"maas-api-revoke-sub-1b92a539-pmxjk\" (UID: \"f7115d25-0f2d-4679-958b-d31e313b197f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk" Oct 30 22:03:28.218078 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.217946 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7115d25-0f2d-4679-958b-d31e313b197f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b92a539-pmxjk\" (UID: \"f7115d25-0f2d-4679-958b-d31e313b197f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk" Oct 30 22:03:28.218078 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.218034 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7115d25-0f2d-4679-958b-d31e313b197f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b92a539-pmxjk\" (UID: \"f7115d25-0f2d-4679-958b-d31e313b197f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk" Oct 30 22:03:28.230806 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.230785 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:28.231978 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.231958 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk"] Oct 30 22:03:28.276513 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.276492 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:28.276513 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.276492 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:28.276730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.276561 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:03:28.319033 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.319006 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59dc\" (UniqueName: \"kubernetes.io/projected/f7115d25-0f2d-4679-958b-d31e313b197f-kube-api-access-l59dc\") pod \"maas-api-revoke-sub-1b92a539-pmxjk\" (UID: \"f7115d25-0f2d-4679-958b-d31e313b197f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk" Oct 30 22:03:28.319146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.319042 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7115d25-0f2d-4679-958b-d31e313b197f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b92a539-pmxjk\" (UID: \"f7115d25-0f2d-4679-958b-d31e313b197f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk" Oct 30 22:03:28.319146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.319107 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7115d25-0f2d-4679-958b-d31e313b197f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b92a539-pmxjk\" (UID: \"f7115d25-0f2d-4679-958b-d31e313b197f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk" Oct 30 22:03:28.319772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.319754 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7115d25-0f2d-4679-958b-d31e313b197f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b92a539-pmxjk\" (UID: \"f7115d25-0f2d-4679-958b-d31e313b197f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk" Oct 30 22:03:28.321433 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.321409 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7115d25-0f2d-4679-958b-d31e313b197f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b92a539-pmxjk\" (UID: \"f7115d25-0f2d-4679-958b-d31e313b197f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk" Oct 30 22:03:28.380148 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.380129 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l59dc\" (UniqueName: \"kubernetes.io/projected/f7115d25-0f2d-4679-958b-d31e313b197f-kube-api-access-l59dc\") pod \"maas-api-revoke-sub-1b92a539-pmxjk\" (UID: \"f7115d25-0f2d-4679-958b-d31e313b197f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk" Oct 30 22:03:28.493004 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.492929 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk" Oct 30 22:03:28.870929 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:28.870890 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk"] Oct 30 22:03:28.871528 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:03:28.871501 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7115d25_0f2d_4679_958b_d31e313b197f.slice/crio-81513f1a61080aa2e2c9c700c3d69e4877b683853c4f6e3e587a30b9d146f269 WatchSource:0}: Error finding container 81513f1a61080aa2e2c9c700c3d69e4877b683853c4f6e3e587a30b9d146f269: Status 404 returned error can't find the container with id 81513f1a61080aa2e2c9c700c3d69e4877b683853c4f6e3e587a30b9d146f269 Oct 30 22:03:29.467196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.467124 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk" event={"ID":"f7115d25-0f2d-4679-958b-d31e313b197f","Type":"ContainerStarted","Data":"f9b191e4bc29ffa127ea6a8ece309f7f013efedf468bfca4e2ba9ee6384b2615"} Oct 30 22:03:29.467196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.467172 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk" event={"ID":"f7115d25-0f2d-4679-958b-d31e313b197f","Type":"ContainerStarted","Data":"81513f1a61080aa2e2c9c700c3d69e4877b683853c4f6e3e587a30b9d146f269"} Oct 30 22:03:29.584000 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.583949 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk" podStartSLOduration=1.583937096 podStartE2EDuration="1.583937096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:29.580910699 +0000 UTC m=+1142.681342431" watchObservedRunningTime="2026-10-30 22:03:29.583937096 +0000 UTC m=+1142.684368828" Oct 30 22:03:29.786056 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.786031 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4"] Oct 30 22:03:29.789998 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.789981 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.834722 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.834697 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-envoy\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.834830 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.834738 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-workload-certs\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.834830 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.834756 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfx2k\" (UniqueName: \"kubernetes.io/projected/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-kube-api-access-cfx2k\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.834945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.834899 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-token\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.834989 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.834942 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-credential-socket\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.835046 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.835034 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-podinfo\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.835081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.835073 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-data\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.835123 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.835111 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-workload-socket\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.835164 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.835129 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istiod-ca-cert\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.841958 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.841936 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-536975-openshift-default-dockercfg-xkvs2\"" Oct 30 22:03:29.850399 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.850377 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4"] Oct 30 22:03:29.936075 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.936053 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-data\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.936184 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.936095 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-workload-socket\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.936184 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.936114 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istiod-ca-cert\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.936184 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.936156 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-envoy\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.936337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.936181 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-workload-certs\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.936337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.936210 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfx2k\" (UniqueName: \"kubernetes.io/projected/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-kube-api-access-cfx2k\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.936337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.936249 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-token\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.936337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.936280 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-credential-socket\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.936337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.936333 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-podinfo\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.936582 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.936470 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-workload-socket\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.936582 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.936502 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-data\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.936727 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.936650 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-workload-certs\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.936832 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.936795 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-credential-socket\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.936932 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.936883 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istiod-ca-cert\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.938539 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.938519 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-envoy\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.938930 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.938908 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-podinfo\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.997144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.997117 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-token\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:29.997252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:29.997238 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfx2k\" (UniqueName: \"kubernetes.io/projected/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-kube-api-access-cfx2k\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:30.132339 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:30.132318 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:30.476116 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:30.476041 4335 generic.go:356] "Generic (PLEG): container finished" podID="f7115d25-0f2d-4679-958b-d31e313b197f" containerID="f9b191e4bc29ffa127ea6a8ece309f7f013efedf468bfca4e2ba9ee6384b2615" exitCode=0 Oct 30 22:03:30.476116 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:30.476110 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk" event={"ID":"f7115d25-0f2d-4679-958b-d31e313b197f","Type":"ContainerDied","Data":"f9b191e4bc29ffa127ea6a8ece309f7f013efedf468bfca4e2ba9ee6384b2615"} Oct 30 22:03:30.565305 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:30.565279 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4"] Oct 30 22:03:30.565482 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:03:30.565459 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b3ae09_c03e_4c8a_95ea_31140419aa9a.slice/crio-40fcc84f4cb7578a687f914b465e0039f6da256ecd4906dcb09c825e97e0c188 WatchSource:0}: Error finding container 40fcc84f4cb7578a687f914b465e0039f6da256ecd4906dcb09c825e97e0c188: Status 404 returned error can't find the container with id 40fcc84f4cb7578a687f914b465e0039f6da256ecd4906dcb09c825e97e0c188 Oct 30 22:03:30.567940 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:30.567908 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:03:30.568016 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:30.567984 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:03:30.568050 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:30.568028 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:03:31.484807 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:31.484772 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" event={"ID":"a3b3ae09-c03e-4c8a-95ea-31140419aa9a","Type":"ContainerStarted","Data":"6b103838b61676786b960148cbf1d5da2fead36d8e623199b8d032f1514e0f38"} Oct 30 22:03:31.485207 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:31.484821 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" event={"ID":"a3b3ae09-c03e-4c8a-95ea-31140419aa9a","Type":"ContainerStarted","Data":"40fcc84f4cb7578a687f914b465e0039f6da256ecd4906dcb09c825e97e0c188"} Oct 30 22:03:31.655197 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:31.655137 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" podStartSLOduration=2.655119889 podStartE2EDuration="2.655119889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:31.654606437 +0000 UTC m=+1144.755038168" watchObservedRunningTime="2026-10-30 22:03:31.655119889 +0000 UTC m=+1144.755551623" Oct 30 22:03:31.675403 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:31.675379 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk" Oct 30 22:03:31.757200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:31.757124 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7115d25-0f2d-4679-958b-d31e313b197f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7115d25-0f2d-4679-958b-d31e313b197f-maas-api-openshift-ca\") pod \"f7115d25-0f2d-4679-958b-d31e313b197f\" (UID: \"f7115d25-0f2d-4679-958b-d31e313b197f\") " Oct 30 22:03:31.757358 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:31.757223 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7115d25-0f2d-4679-958b-d31e313b197f-kube-api-access-l59dc\" (UniqueName: \"kubernetes.io/projected/f7115d25-0f2d-4679-958b-d31e313b197f-kube-api-access-l59dc\") pod \"f7115d25-0f2d-4679-958b-d31e313b197f\" (UID: \"f7115d25-0f2d-4679-958b-d31e313b197f\") " Oct 30 22:03:31.757358 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:31.757253 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7115d25-0f2d-4679-958b-d31e313b197f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7115d25-0f2d-4679-958b-d31e313b197f-maas-api-xks-ca\") pod \"f7115d25-0f2d-4679-958b-d31e313b197f\" (UID: \"f7115d25-0f2d-4679-958b-d31e313b197f\") " Oct 30 22:03:31.757456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:31.757425 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7115d25-0f2d-4679-958b-d31e313b197f-maas-api-openshift-ca" pod "f7115d25-0f2d-4679-958b-d31e313b197f" (UID: "f7115d25-0f2d-4679-958b-d31e313b197f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:31.757729 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:31.757705 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7115d25-0f2d-4679-958b-d31e313b197f-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:31.759394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:31.759366 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7115d25-0f2d-4679-958b-d31e313b197f-maas-api-xks-ca" pod "f7115d25-0f2d-4679-958b-d31e313b197f" (UID: "f7115d25-0f2d-4679-958b-d31e313b197f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:31.759531 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:31.759517 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7115d25-0f2d-4679-958b-d31e313b197f-kube-api-access-l59dc" pod "f7115d25-0f2d-4679-958b-d31e313b197f" (UID: "f7115d25-0f2d-4679-958b-d31e313b197f"). InnerVolumeSpecName "kube-api-access-l59dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:31.859194 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:31.859158 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l59dc\" (UniqueName: \"kubernetes.io/projected/f7115d25-0f2d-4679-958b-d31e313b197f-kube-api-access-l59dc\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:31.859194 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:31.859186 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7115d25-0f2d-4679-958b-d31e313b197f-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:32.132706 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.132662 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:32.134312 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.134289 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:03:32.134425 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.134333 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" podUID="a3b3ae09-c03e-4c8a-95ea-31140419aa9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 22:03:32.493434 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.493362 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk" Oct 30 22:03:32.515991 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.515960 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk" event={"ID":"f7115d25-0f2d-4679-958b-d31e313b197f","Type":"ContainerDied","Data":"81513f1a61080aa2e2c9c700c3d69e4877b683853c4f6e3e587a30b9d146f269"} Oct 30 22:03:32.516235 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.516217 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81513f1a61080aa2e2c9c700c3d69e4877b683853c4f6e3e587a30b9d146f269" Oct 30 22:03:32.657575 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.657543 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7"] Oct 30 22:03:32.658086 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.658073 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f7115d25-0f2d-4679-958b-d31e313b197f" containerName="revoke-keys" Oct 30 22:03:32.658132 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.658088 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7115d25-0f2d-4679-958b-d31e313b197f" containerName="revoke-keys" Oct 30 22:03:32.658213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.658204 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7115d25-0f2d-4679-958b-d31e313b197f" containerName="revoke-keys" Oct 30 22:03:32.661837 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.661822 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7" Oct 30 22:03:32.721640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.717775 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:32.721640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.717900 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:32.721640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.717968 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:32.721640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.718101 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:03:32.737779 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.737752 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7"] Oct 30 22:03:32.768510 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.768445 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b97nv\" (UniqueName: \"kubernetes.io/projected/ddf30afc-138c-4265-aadd-1193e3744df2-kube-api-access-b97nv\") pod \"maas-api-revoke-sub-faf53655-gvvr7\" (UID: \"ddf30afc-138c-4265-aadd-1193e3744df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7" Oct 30 22:03:32.768705 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.768569 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddf30afc-138c-4265-aadd-1193e3744df2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faf53655-gvvr7\" (UID: \"ddf30afc-138c-4265-aadd-1193e3744df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7" Oct 30 22:03:32.768767 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.768750 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddf30afc-138c-4265-aadd-1193e3744df2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faf53655-gvvr7\" (UID: \"ddf30afc-138c-4265-aadd-1193e3744df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7" Oct 30 22:03:32.869793 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.869762 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b97nv\" (UniqueName: \"kubernetes.io/projected/ddf30afc-138c-4265-aadd-1193e3744df2-kube-api-access-b97nv\") pod \"maas-api-revoke-sub-faf53655-gvvr7\" (UID: \"ddf30afc-138c-4265-aadd-1193e3744df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7" Oct 30 22:03:32.869941 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.869853 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddf30afc-138c-4265-aadd-1193e3744df2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faf53655-gvvr7\" (UID: \"ddf30afc-138c-4265-aadd-1193e3744df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7" Oct 30 22:03:32.869998 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.869972 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddf30afc-138c-4265-aadd-1193e3744df2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faf53655-gvvr7\" (UID: \"ddf30afc-138c-4265-aadd-1193e3744df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7" Oct 30 22:03:32.870750 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.870724 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddf30afc-138c-4265-aadd-1193e3744df2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faf53655-gvvr7\" (UID: \"ddf30afc-138c-4265-aadd-1193e3744df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7" Oct 30 22:03:32.872309 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.872285 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddf30afc-138c-4265-aadd-1193e3744df2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faf53655-gvvr7\" (UID: \"ddf30afc-138c-4265-aadd-1193e3744df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7" Oct 30 22:03:32.930644 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.930588 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b97nv\" (UniqueName: \"kubernetes.io/projected/ddf30afc-138c-4265-aadd-1193e3744df2-kube-api-access-b97nv\") pod \"maas-api-revoke-sub-faf53655-gvvr7\" (UID: \"ddf30afc-138c-4265-aadd-1193e3744df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7" Oct 30 22:03:32.975584 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:32.975563 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7" Oct 30 22:03:33.133490 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:33.133454 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:03:33.133669 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:33.133512 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" podUID="a3b3ae09-c03e-4c8a-95ea-31140419aa9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 22:03:33.380475 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:33.380447 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7"] Oct 30 22:03:33.380911 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:03:33.380888 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf30afc_138c_4265_aadd_1193e3744df2.slice/crio-ba70f4daf26a270b4888f5577cbca95e106b0a05b378ac7c29d764663b65da84 WatchSource:0}: Error finding container ba70f4daf26a270b4888f5577cbca95e106b0a05b378ac7c29d764663b65da84: Status 404 returned error can't find the container with id ba70f4daf26a270b4888f5577cbca95e106b0a05b378ac7c29d764663b65da84 Oct 30 22:03:33.525796 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:33.525761 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7" event={"ID":"ddf30afc-138c-4265-aadd-1193e3744df2","Type":"ContainerStarted","Data":"ba70f4daf26a270b4888f5577cbca95e106b0a05b378ac7c29d764663b65da84"} Oct 30 22:03:34.133539 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:34.133493 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:03:34.133720 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:34.133559 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" podUID="a3b3ae09-c03e-4c8a-95ea-31140419aa9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 22:03:34.535346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:34.535316 4335 generic.go:356] "Generic (PLEG): container finished" podID="ddf30afc-138c-4265-aadd-1193e3744df2" containerID="fc8f57d2908ec6717dec492a241bc5114877830363686518511698f9dc8ef6f4" exitCode=0 Oct 30 22:03:34.535816 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:34.535382 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7" event={"ID":"ddf30afc-138c-4265-aadd-1193e3744df2","Type":"ContainerDied","Data":"fc8f57d2908ec6717dec492a241bc5114877830363686518511698f9dc8ef6f4"} Oct 30 22:03:35.133203 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:35.133165 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:03:35.133401 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:35.133223 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" podUID="a3b3ae09-c03e-4c8a-95ea-31140419aa9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 22:03:35.749686 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:35.749651 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7" Oct 30 22:03:35.828327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:35.828293 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddf30afc-138c-4265-aadd-1193e3744df2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddf30afc-138c-4265-aadd-1193e3744df2-maas-api-xks-ca\") pod \"ddf30afc-138c-4265-aadd-1193e3744df2\" (UID: \"ddf30afc-138c-4265-aadd-1193e3744df2\") " Oct 30 22:03:35.828478 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:35.828427 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddf30afc-138c-4265-aadd-1193e3744df2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddf30afc-138c-4265-aadd-1193e3744df2-maas-api-openshift-ca\") pod \"ddf30afc-138c-4265-aadd-1193e3744df2\" (UID: \"ddf30afc-138c-4265-aadd-1193e3744df2\") " Oct 30 22:03:35.828562 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:35.828506 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddf30afc-138c-4265-aadd-1193e3744df2-kube-api-access-b97nv\" (UniqueName: \"kubernetes.io/projected/ddf30afc-138c-4265-aadd-1193e3744df2-kube-api-access-b97nv\") pod \"ddf30afc-138c-4265-aadd-1193e3744df2\" (UID: \"ddf30afc-138c-4265-aadd-1193e3744df2\") " Oct 30 22:03:35.828858 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:35.828733 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf30afc-138c-4265-aadd-1193e3744df2-maas-api-openshift-ca" pod "ddf30afc-138c-4265-aadd-1193e3744df2" (UID: "ddf30afc-138c-4265-aadd-1193e3744df2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:35.828993 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:35.828977 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddf30afc-138c-4265-aadd-1193e3744df2-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:35.830838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:35.830806 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf30afc-138c-4265-aadd-1193e3744df2-kube-api-access-b97nv" pod "ddf30afc-138c-4265-aadd-1193e3744df2" (UID: "ddf30afc-138c-4265-aadd-1193e3744df2"). InnerVolumeSpecName "kube-api-access-b97nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:35.830929 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:35.830871 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf30afc-138c-4265-aadd-1193e3744df2-maas-api-xks-ca" pod "ddf30afc-138c-4265-aadd-1193e3744df2" (UID: "ddf30afc-138c-4265-aadd-1193e3744df2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:35.930397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:35.930356 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddf30afc-138c-4265-aadd-1193e3744df2-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:35.930397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:35.930393 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b97nv\" (UniqueName: \"kubernetes.io/projected/ddf30afc-138c-4265-aadd-1193e3744df2-kube-api-access-b97nv\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:36.132768 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:36.132670 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:03:36.132768 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:36.132746 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" podUID="a3b3ae09-c03e-4c8a-95ea-31140419aa9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 22:03:36.552149 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:36.552113 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7" event={"ID":"ddf30afc-138c-4265-aadd-1193e3744df2","Type":"ContainerDied","Data":"ba70f4daf26a270b4888f5577cbca95e106b0a05b378ac7c29d764663b65da84"} Oct 30 22:03:36.552314 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:36.552179 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba70f4daf26a270b4888f5577cbca95e106b0a05b378ac7c29d764663b65da84" Oct 30 22:03:36.552314 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:36.552132 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7" Oct 30 22:03:37.214001 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:37.213972 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:37.214667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:37.214649 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:37.215311 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:37.215290 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:03:39.058677 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.058649 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn"] Oct 30 22:03:39.059249 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.059232 4335 cpu_manager.go:414] "containerMap: removing container" podUID="ddf30afc-138c-4265-aadd-1193e3744df2" containerName="revoke-keys" Oct 30 22:03:39.059307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.059251 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddf30afc-138c-4265-aadd-1193e3744df2" containerName="revoke-keys" Oct 30 22:03:39.059409 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.059398 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddf30afc-138c-4265-aadd-1193e3744df2" containerName="revoke-keys" Oct 30 22:03:39.062872 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.062854 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn" Oct 30 22:03:39.120936 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.120909 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn"] Oct 30 22:03:39.162835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.162808 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1faa26c2-e6db-41c6-8598-a296df151cea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57b29051-562mn\" (UID: \"1faa26c2-e6db-41c6-8598-a296df151cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn" Oct 30 22:03:39.162992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.162864 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb8gh\" (UniqueName: \"kubernetes.io/projected/1faa26c2-e6db-41c6-8598-a296df151cea-kube-api-access-lb8gh\") pod \"maas-api-revoke-sub-57b29051-562mn\" (UID: \"1faa26c2-e6db-41c6-8598-a296df151cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn" Oct 30 22:03:39.163054 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.163032 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1faa26c2-e6db-41c6-8598-a296df151cea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57b29051-562mn\" (UID: \"1faa26c2-e6db-41c6-8598-a296df151cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn" Oct 30 22:03:39.165393 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.165368 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:39.165393 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.165377 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:39.165549 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.165398 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:39.165549 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.165433 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:03:39.263791 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.263766 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1faa26c2-e6db-41c6-8598-a296df151cea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57b29051-562mn\" (UID: \"1faa26c2-e6db-41c6-8598-a296df151cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn" Oct 30 22:03:39.263925 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.263803 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb8gh\" (UniqueName: \"kubernetes.io/projected/1faa26c2-e6db-41c6-8598-a296df151cea-kube-api-access-lb8gh\") pod \"maas-api-revoke-sub-57b29051-562mn\" (UID: \"1faa26c2-e6db-41c6-8598-a296df151cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn" Oct 30 22:03:39.263925 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.263867 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1faa26c2-e6db-41c6-8598-a296df151cea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57b29051-562mn\" (UID: \"1faa26c2-e6db-41c6-8598-a296df151cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn" Oct 30 22:03:39.264529 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.264503 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1faa26c2-e6db-41c6-8598-a296df151cea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57b29051-562mn\" (UID: \"1faa26c2-e6db-41c6-8598-a296df151cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn" Oct 30 22:03:39.266337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.266315 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1faa26c2-e6db-41c6-8598-a296df151cea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57b29051-562mn\" (UID: \"1faa26c2-e6db-41c6-8598-a296df151cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn" Oct 30 22:03:39.324456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.324405 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb8gh\" (UniqueName: \"kubernetes.io/projected/1faa26c2-e6db-41c6-8598-a296df151cea-kube-api-access-lb8gh\") pod \"maas-api-revoke-sub-57b29051-562mn\" (UID: \"1faa26c2-e6db-41c6-8598-a296df151cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn" Oct 30 22:03:39.378594 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.378569 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn" Oct 30 22:03:39.762058 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:03:39.762030 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1faa26c2_e6db_41c6_8598_a296df151cea.slice/crio-ad9d1202bf94db5e9db12869b8bf40e75b864bfc5fee5baf9447a097fa7e4815 WatchSource:0}: Error finding container ad9d1202bf94db5e9db12869b8bf40e75b864bfc5fee5baf9447a097fa7e4815: Status 404 returned error can't find the container with id ad9d1202bf94db5e9db12869b8bf40e75b864bfc5fee5baf9447a097fa7e4815 Oct 30 22:03:39.762936 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:39.762917 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn"] Oct 30 22:03:40.584158 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:40.584125 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn" event={"ID":"1faa26c2-e6db-41c6-8598-a296df151cea","Type":"ContainerStarted","Data":"070039490cf57217a10b494340f0b2155542e106ebab3bfd0579a0c9546b6c4c"} Oct 30 22:03:40.584552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:40.584185 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn" event={"ID":"1faa26c2-e6db-41c6-8598-a296df151cea","Type":"ContainerStarted","Data":"ad9d1202bf94db5e9db12869b8bf40e75b864bfc5fee5baf9447a097fa7e4815"} Oct 30 22:03:40.700242 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:40.700189 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn" podStartSLOduration=1.700171412 podStartE2EDuration="1.700171412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:40.699872364 +0000 UTC m=+1153.800304099" watchObservedRunningTime="2026-10-30 22:03:40.700171412 +0000 UTC m=+1153.800603145" Oct 30 22:03:41.593757 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:41.593722 4335 generic.go:356] "Generic (PLEG): container finished" podID="1faa26c2-e6db-41c6-8598-a296df151cea" containerID="070039490cf57217a10b494340f0b2155542e106ebab3bfd0579a0c9546b6c4c" exitCode=0 Oct 30 22:03:41.594174 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:41.593803 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn" event={"ID":"1faa26c2-e6db-41c6-8598-a296df151cea","Type":"ContainerDied","Data":"070039490cf57217a10b494340f0b2155542e106ebab3bfd0579a0c9546b6c4c"} Oct 30 22:03:42.739709 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:42.739680 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn" Oct 30 22:03:42.899542 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:42.899469 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1faa26c2-e6db-41c6-8598-a296df151cea-kube-api-access-lb8gh\" (UniqueName: \"kubernetes.io/projected/1faa26c2-e6db-41c6-8598-a296df151cea-kube-api-access-lb8gh\") pod \"1faa26c2-e6db-41c6-8598-a296df151cea\" (UID: \"1faa26c2-e6db-41c6-8598-a296df151cea\") " Oct 30 22:03:42.899542 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:42.899524 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1faa26c2-e6db-41c6-8598-a296df151cea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1faa26c2-e6db-41c6-8598-a296df151cea-maas-api-xks-ca\") pod \"1faa26c2-e6db-41c6-8598-a296df151cea\" (UID: \"1faa26c2-e6db-41c6-8598-a296df151cea\") " Oct 30 22:03:42.899810 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:42.899662 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1faa26c2-e6db-41c6-8598-a296df151cea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1faa26c2-e6db-41c6-8598-a296df151cea-maas-api-openshift-ca\") pod \"1faa26c2-e6db-41c6-8598-a296df151cea\" (UID: \"1faa26c2-e6db-41c6-8598-a296df151cea\") " Oct 30 22:03:42.900317 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:42.900288 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1faa26c2-e6db-41c6-8598-a296df151cea-maas-api-openshift-ca" pod "1faa26c2-e6db-41c6-8598-a296df151cea" (UID: "1faa26c2-e6db-41c6-8598-a296df151cea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:42.902075 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:42.902051 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1faa26c2-e6db-41c6-8598-a296df151cea-maas-api-xks-ca" pod "1faa26c2-e6db-41c6-8598-a296df151cea" (UID: "1faa26c2-e6db-41c6-8598-a296df151cea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:42.902550 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:42.902529 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1faa26c2-e6db-41c6-8598-a296df151cea-kube-api-access-lb8gh" pod "1faa26c2-e6db-41c6-8598-a296df151cea" (UID: "1faa26c2-e6db-41c6-8598-a296df151cea"). InnerVolumeSpecName "kube-api-access-lb8gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:43.001440 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:43.001397 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb8gh\" (UniqueName: \"kubernetes.io/projected/1faa26c2-e6db-41c6-8598-a296df151cea-kube-api-access-lb8gh\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:43.001553 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:43.001443 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1faa26c2-e6db-41c6-8598-a296df151cea-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:43.001553 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:43.001459 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1faa26c2-e6db-41c6-8598-a296df151cea-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:43.612849 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:43.612819 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn" Oct 30 22:03:43.612849 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:43.612841 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn" event={"ID":"1faa26c2-e6db-41c6-8598-a296df151cea","Type":"ContainerDied","Data":"ad9d1202bf94db5e9db12869b8bf40e75b864bfc5fee5baf9447a097fa7e4815"} Oct 30 22:03:43.613073 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:43.612898 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad9d1202bf94db5e9db12869b8bf40e75b864bfc5fee5baf9447a097fa7e4815" Oct 30 22:03:45.283804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.283775 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58"] Oct 30 22:03:45.284344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.284331 4335 cpu_manager.go:414] "containerMap: removing container" podUID="1faa26c2-e6db-41c6-8598-a296df151cea" containerName="revoke-keys" Oct 30 22:03:45.284396 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.284345 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1faa26c2-e6db-41c6-8598-a296df151cea" containerName="revoke-keys" Oct 30 22:03:45.284449 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.284440 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="1faa26c2-e6db-41c6-8598-a296df151cea" containerName="revoke-keys" Oct 30 22:03:45.287884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.287861 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58" Oct 30 22:03:45.340479 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.340449 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:45.340604 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.340581 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:45.340749 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.340733 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:45.389936 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.389915 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:03:45.400813 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.399327 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58"] Oct 30 22:03:45.428955 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.428929 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddaab237-ddfc-4371-8288-49752bd1529e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62119c3d-tjp58\" (UID: \"ddaab237-ddfc-4371-8288-49752bd1529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58" Oct 30 22:03:45.429086 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.429066 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76hqp\" (UniqueName: \"kubernetes.io/projected/ddaab237-ddfc-4371-8288-49752bd1529e-kube-api-access-76hqp\") pod \"maas-api-revoke-sub-62119c3d-tjp58\" (UID: \"ddaab237-ddfc-4371-8288-49752bd1529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58" Oct 30 22:03:45.429129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.429109 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddaab237-ddfc-4371-8288-49752bd1529e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62119c3d-tjp58\" (UID: \"ddaab237-ddfc-4371-8288-49752bd1529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58" Oct 30 22:03:45.530009 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.529983 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76hqp\" (UniqueName: \"kubernetes.io/projected/ddaab237-ddfc-4371-8288-49752bd1529e-kube-api-access-76hqp\") pod \"maas-api-revoke-sub-62119c3d-tjp58\" (UID: \"ddaab237-ddfc-4371-8288-49752bd1529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58" Oct 30 22:03:45.530117 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.530021 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddaab237-ddfc-4371-8288-49752bd1529e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62119c3d-tjp58\" (UID: \"ddaab237-ddfc-4371-8288-49752bd1529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58" Oct 30 22:03:45.530117 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.530086 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddaab237-ddfc-4371-8288-49752bd1529e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62119c3d-tjp58\" (UID: \"ddaab237-ddfc-4371-8288-49752bd1529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58" Oct 30 22:03:45.530737 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.530718 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddaab237-ddfc-4371-8288-49752bd1529e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62119c3d-tjp58\" (UID: \"ddaab237-ddfc-4371-8288-49752bd1529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58" Oct 30 22:03:45.532494 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.532468 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddaab237-ddfc-4371-8288-49752bd1529e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62119c3d-tjp58\" (UID: \"ddaab237-ddfc-4371-8288-49752bd1529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58" Oct 30 22:03:45.567151 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.567129 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:03:45.590674 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.590651 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76hqp\" (UniqueName: \"kubernetes.io/projected/ddaab237-ddfc-4371-8288-49752bd1529e-kube-api-access-76hqp\") pod \"maas-api-revoke-sub-62119c3d-tjp58\" (UID: \"ddaab237-ddfc-4371-8288-49752bd1529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58" Oct 30 22:03:45.601136 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.601112 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58" Oct 30 22:03:45.983702 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:45.983675 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58"] Oct 30 22:03:45.985012 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:03:45.984981 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddaab237_ddfc_4371_8288_49752bd1529e.slice/crio-ce6de8e0d49e3f5c63e64675b7a4dfa48bf45cab604529b1d7a5e4f6aa6df8e2 WatchSource:0}: Error finding container ce6de8e0d49e3f5c63e64675b7a4dfa48bf45cab604529b1d7a5e4f6aa6df8e2: Status 404 returned error can't find the container with id ce6de8e0d49e3f5c63e64675b7a4dfa48bf45cab604529b1d7a5e4f6aa6df8e2 Oct 30 22:03:46.641941 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:46.641844 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58" event={"ID":"ddaab237-ddfc-4371-8288-49752bd1529e","Type":"ContainerStarted","Data":"eee024623e7de2c3e50770a268a31bcfa3daced2b57df4dec0d5d92a5715bc5a"} Oct 30 22:03:46.641941 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:46.641909 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58" event={"ID":"ddaab237-ddfc-4371-8288-49752bd1529e","Type":"ContainerStarted","Data":"ce6de8e0d49e3f5c63e64675b7a4dfa48bf45cab604529b1d7a5e4f6aa6df8e2"} Oct 30 22:03:47.650891 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:47.650862 4335 generic.go:356] "Generic (PLEG): container finished" podID="ddaab237-ddfc-4371-8288-49752bd1529e" containerID="eee024623e7de2c3e50770a268a31bcfa3daced2b57df4dec0d5d92a5715bc5a" exitCode=0 Oct 30 22:03:47.651282 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:47.650946 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58" event={"ID":"ddaab237-ddfc-4371-8288-49752bd1529e","Type":"ContainerDied","Data":"eee024623e7de2c3e50770a268a31bcfa3daced2b57df4dec0d5d92a5715bc5a"} Oct 30 22:03:48.817593 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:48.817569 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58" Oct 30 22:03:48.889728 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:48.889703 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddaab237-ddfc-4371-8288-49752bd1529e-kube-api-access-76hqp\" (UniqueName: \"kubernetes.io/projected/ddaab237-ddfc-4371-8288-49752bd1529e-kube-api-access-76hqp\") pod \"ddaab237-ddfc-4371-8288-49752bd1529e\" (UID: \"ddaab237-ddfc-4371-8288-49752bd1529e\") " Oct 30 22:03:48.889921 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:48.889904 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddaab237-ddfc-4371-8288-49752bd1529e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddaab237-ddfc-4371-8288-49752bd1529e-maas-api-xks-ca\") pod \"ddaab237-ddfc-4371-8288-49752bd1529e\" (UID: \"ddaab237-ddfc-4371-8288-49752bd1529e\") " Oct 30 22:03:48.889967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:48.889947 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddaab237-ddfc-4371-8288-49752bd1529e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddaab237-ddfc-4371-8288-49752bd1529e-maas-api-openshift-ca\") pod \"ddaab237-ddfc-4371-8288-49752bd1529e\" (UID: \"ddaab237-ddfc-4371-8288-49752bd1529e\") " Oct 30 22:03:48.890563 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:48.890526 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddaab237-ddfc-4371-8288-49752bd1529e-maas-api-openshift-ca" pod "ddaab237-ddfc-4371-8288-49752bd1529e" (UID: "ddaab237-ddfc-4371-8288-49752bd1529e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:48.892034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:48.892009 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddaab237-ddfc-4371-8288-49752bd1529e-maas-api-xks-ca" pod "ddaab237-ddfc-4371-8288-49752bd1529e" (UID: "ddaab237-ddfc-4371-8288-49752bd1529e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:48.892034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:48.892008 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddaab237-ddfc-4371-8288-49752bd1529e-kube-api-access-76hqp" pod "ddaab237-ddfc-4371-8288-49752bd1529e" (UID: "ddaab237-ddfc-4371-8288-49752bd1529e"). InnerVolumeSpecName "kube-api-access-76hqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:48.991855 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:48.991794 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddaab237-ddfc-4371-8288-49752bd1529e-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.991855 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:48.991818 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddaab237-ddfc-4371-8288-49752bd1529e-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.991855 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:48.991828 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76hqp\" (UniqueName: \"kubernetes.io/projected/ddaab237-ddfc-4371-8288-49752bd1529e-kube-api-access-76hqp\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:49.667250 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:49.667219 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58" Oct 30 22:03:49.667424 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:49.667267 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58" event={"ID":"ddaab237-ddfc-4371-8288-49752bd1529e","Type":"ContainerDied","Data":"ce6de8e0d49e3f5c63e64675b7a4dfa48bf45cab604529b1d7a5e4f6aa6df8e2"} Oct 30 22:03:49.667424 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:49.667346 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce6de8e0d49e3f5c63e64675b7a4dfa48bf45cab604529b1d7a5e4f6aa6df8e2" Oct 30 22:03:55.320895 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:03:55.320862 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:03:58.893823 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:58.893790 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj"] Oct 30 22:03:58.894357 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:58.894343 4335 cpu_manager.go:414] "containerMap: removing container" podUID="ddaab237-ddfc-4371-8288-49752bd1529e" containerName="revoke-keys" Oct 30 22:03:58.894420 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:58.894357 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddaab237-ddfc-4371-8288-49752bd1529e" containerName="revoke-keys" Oct 30 22:03:58.894471 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:58.894462 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddaab237-ddfc-4371-8288-49752bd1529e" containerName="revoke-keys" Oct 30 22:03:58.898161 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:58.898147 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj" Oct 30 22:03:58.951274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:58.951236 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:58.990217 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:58.990187 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z56p9\" (UniqueName: \"kubernetes.io/projected/c82efd63-6d58-424c-bcff-772354c678f9-kube-api-access-z56p9\") pod \"maas-api-revoke-sub-09d0818d-rpstj\" (UID: \"c82efd63-6d58-424c-bcff-772354c678f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj" Oct 30 22:03:58.990370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:58.990225 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82efd63-6d58-424c-bcff-772354c678f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09d0818d-rpstj\" (UID: \"c82efd63-6d58-424c-bcff-772354c678f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj" Oct 30 22:03:58.990370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:58.990274 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82efd63-6d58-424c-bcff-772354c678f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09d0818d-rpstj\" (UID: \"c82efd63-6d58-424c-bcff-772354c678f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj" Oct 30 22:03:59.000772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:59.000749 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:59.000772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:59.000749 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:59.000971 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:59.000892 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:03:59.005412 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:59.005389 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj"] Oct 30 22:03:59.091381 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:59.091352 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z56p9\" (UniqueName: \"kubernetes.io/projected/c82efd63-6d58-424c-bcff-772354c678f9-kube-api-access-z56p9\") pod \"maas-api-revoke-sub-09d0818d-rpstj\" (UID: \"c82efd63-6d58-424c-bcff-772354c678f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj" Oct 30 22:03:59.091381 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:59.091386 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82efd63-6d58-424c-bcff-772354c678f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09d0818d-rpstj\" (UID: \"c82efd63-6d58-424c-bcff-772354c678f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj" Oct 30 22:03:59.091606 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:59.091434 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82efd63-6d58-424c-bcff-772354c678f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09d0818d-rpstj\" (UID: \"c82efd63-6d58-424c-bcff-772354c678f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj" Oct 30 22:03:59.092133 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:59.092113 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82efd63-6d58-424c-bcff-772354c678f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09d0818d-rpstj\" (UID: \"c82efd63-6d58-424c-bcff-772354c678f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj" Oct 30 22:03:59.093958 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:59.093939 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82efd63-6d58-424c-bcff-772354c678f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09d0818d-rpstj\" (UID: \"c82efd63-6d58-424c-bcff-772354c678f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj" Oct 30 22:03:59.152418 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:59.152356 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z56p9\" (UniqueName: \"kubernetes.io/projected/c82efd63-6d58-424c-bcff-772354c678f9-kube-api-access-z56p9\") pod \"maas-api-revoke-sub-09d0818d-rpstj\" (UID: \"c82efd63-6d58-424c-bcff-772354c678f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj" Oct 30 22:03:59.211146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:59.211118 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj" Oct 30 22:03:59.591808 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:59.591781 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj"] Oct 30 22:03:59.592193 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:03:59.592159 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82efd63_6d58_424c_bcff_772354c678f9.slice/crio-7cb8b70d3cb09158c17d0d728b88dce459d4560cd25f79902c00aa5c2995d388 WatchSource:0}: Error finding container 7cb8b70d3cb09158c17d0d728b88dce459d4560cd25f79902c00aa5c2995d388: Status 404 returned error can't find the container with id 7cb8b70d3cb09158c17d0d728b88dce459d4560cd25f79902c00aa5c2995d388 Oct 30 22:03:59.739328 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:03:59.739292 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj" event={"ID":"c82efd63-6d58-424c-bcff-772354c678f9","Type":"ContainerStarted","Data":"7cb8b70d3cb09158c17d0d728b88dce459d4560cd25f79902c00aa5c2995d388"} Oct 30 22:04:00.252810 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.252737 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79"] Oct 30 22:04:00.256881 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.256865 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.306108 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.306079 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-istio-envoy\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.306223 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.306208 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr927\" (UniqueName: \"kubernetes.io/projected/c4940530-3248-4c16-b8b0-5959219c8aca-kube-api-access-gr927\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.306276 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.306233 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-workload-socket\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.306313 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.306284 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4940530-3248-4c16-b8b0-5959219c8aca-istiod-ca-cert\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.306396 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.306384 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4940530-3248-4c16-b8b0-5959219c8aca-istio-token\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.306441 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.306429 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-credential-socket\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.306473 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.306461 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-workload-certs\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.306577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.306561 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-istio-data\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.306609 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.306600 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4940530-3248-4c16-b8b0-5959219c8aca-istio-podinfo\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.309366 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.309346 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-b9fbe2-openshift-default-dockercfg-jrtfd\"" Oct 30 22:04:00.365193 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.365167 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79"] Oct 30 22:04:00.407590 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.407561 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr927\" (UniqueName: \"kubernetes.io/projected/c4940530-3248-4c16-b8b0-5959219c8aca-kube-api-access-gr927\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.407802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.407597 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-workload-socket\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.407802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.407635 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4940530-3248-4c16-b8b0-5959219c8aca-istiod-ca-cert\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.407802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.407685 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4940530-3248-4c16-b8b0-5959219c8aca-istio-token\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.407802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.407721 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-credential-socket\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.407802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.407749 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-workload-certs\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.407802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.407771 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-istio-data\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.407802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.407793 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4940530-3248-4c16-b8b0-5959219c8aca-istio-podinfo\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.408194 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.407823 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-istio-envoy\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.408482 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.408451 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-credential-socket\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.408799 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.408774 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-workload-socket\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.409044 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.409025 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-istio-data\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.409288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.409269 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4940530-3248-4c16-b8b0-5959219c8aca-istiod-ca-cert\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.409561 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.409543 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-workload-certs\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.411548 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.411526 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-istio-envoy\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.411675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.411587 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4940530-3248-4c16-b8b0-5959219c8aca-istio-podinfo\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.468871 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.468840 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4940530-3248-4c16-b8b0-5959219c8aca-istio-token\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.469125 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.469103 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr927\" (UniqueName: \"kubernetes.io/projected/c4940530-3248-4c16-b8b0-5959219c8aca-kube-api-access-gr927\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.571367 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.571342 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:00.750386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.750352 4335 generic.go:356] "Generic (PLEG): container finished" podID="c82efd63-6d58-424c-bcff-772354c678f9" containerID="e459b3223b4cb40b98a1df7fc07df0042f32e56c99633f37095024a269c0281b" exitCode=0 Oct 30 22:04:00.750519 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:00.750392 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj" event={"ID":"c82efd63-6d58-424c-bcff-772354c678f9","Type":"ContainerDied","Data":"e459b3223b4cb40b98a1df7fc07df0042f32e56c99633f37095024a269c0281b"} Oct 30 22:04:01.035947 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:04:01.035914 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4940530_3248_4c16_b8b0_5959219c8aca.slice/crio-ac845655fd4ac5e0f3a4016c7e5dba0c7fc2429a7b8bce7fe62eb9a89ecdde77 WatchSource:0}: Error finding container ac845655fd4ac5e0f3a4016c7e5dba0c7fc2429a7b8bce7fe62eb9a89ecdde77: Status 404 returned error can't find the container with id ac845655fd4ac5e0f3a4016c7e5dba0c7fc2429a7b8bce7fe62eb9a89ecdde77 Oct 30 22:04:01.037717 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:01.037675 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79"] Oct 30 22:04:01.038699 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:01.038659 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:04:01.038815 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:01.038744 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:04:01.038815 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:01.038785 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:04:01.759438 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:01.759392 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" event={"ID":"c4940530-3248-4c16-b8b0-5959219c8aca","Type":"ContainerStarted","Data":"5f0d75698a7ae89ff873e0e405186ecd05120de1a01a6c97e436a1668a78236a"} Oct 30 22:04:01.759975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:01.759488 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" event={"ID":"c4940530-3248-4c16-b8b0-5959219c8aca","Type":"ContainerStarted","Data":"ac845655fd4ac5e0f3a4016c7e5dba0c7fc2429a7b8bce7fe62eb9a89ecdde77"} Oct 30 22:04:01.969168 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:01.969140 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj" Oct 30 22:04:01.985748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:01.985706 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" podStartSLOduration=1.985693909 podStartE2EDuration="1.985693909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:01.933119534 +0000 UTC m=+1175.033551267" watchObservedRunningTime="2026-10-30 22:04:01.985693909 +0000 UTC m=+1175.086125640" Oct 30 22:04:02.039126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:02.039022 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c82efd63-6d58-424c-bcff-772354c678f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82efd63-6d58-424c-bcff-772354c678f9-maas-api-openshift-ca\") pod \"c82efd63-6d58-424c-bcff-772354c678f9\" (UID: \"c82efd63-6d58-424c-bcff-772354c678f9\") " Oct 30 22:04:02.039126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:02.039069 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c82efd63-6d58-424c-bcff-772354c678f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82efd63-6d58-424c-bcff-772354c678f9-maas-api-xks-ca\") pod \"c82efd63-6d58-424c-bcff-772354c678f9\" (UID: \"c82efd63-6d58-424c-bcff-772354c678f9\") " Oct 30 22:04:02.039375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:02.039138 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c82efd63-6d58-424c-bcff-772354c678f9-kube-api-access-z56p9\" (UniqueName: \"kubernetes.io/projected/c82efd63-6d58-424c-bcff-772354c678f9-kube-api-access-z56p9\") pod \"c82efd63-6d58-424c-bcff-772354c678f9\" (UID: \"c82efd63-6d58-424c-bcff-772354c678f9\") " Oct 30 22:04:02.039443 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:02.039410 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82efd63-6d58-424c-bcff-772354c678f9-maas-api-openshift-ca" pod "c82efd63-6d58-424c-bcff-772354c678f9" (UID: "c82efd63-6d58-424c-bcff-772354c678f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:02.039696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:02.039674 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82efd63-6d58-424c-bcff-772354c678f9-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:02.041503 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:02.041469 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82efd63-6d58-424c-bcff-772354c678f9-kube-api-access-z56p9" pod "c82efd63-6d58-424c-bcff-772354c678f9" (UID: "c82efd63-6d58-424c-bcff-772354c678f9"). InnerVolumeSpecName "kube-api-access-z56p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:02.044725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:02.044702 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82efd63-6d58-424c-bcff-772354c678f9-maas-api-xks-ca" pod "c82efd63-6d58-424c-bcff-772354c678f9" (UID: "c82efd63-6d58-424c-bcff-772354c678f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:02.140879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:02.140838 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z56p9\" (UniqueName: \"kubernetes.io/projected/c82efd63-6d58-424c-bcff-772354c678f9-kube-api-access-z56p9\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:02.140879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:02.140881 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82efd63-6d58-424c-bcff-772354c678f9-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:02.572378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:02.572347 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:02.573976 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:02.573951 4335 patch_prober.go:29] interesting pod/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:04:02.574099 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:02.573998 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" podUID="c4940530-3248-4c16-b8b0-5959219c8aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 22:04:02.768209 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:02.768178 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj" Oct 30 22:04:02.768556 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:02.768194 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj" event={"ID":"c82efd63-6d58-424c-bcff-772354c678f9","Type":"ContainerDied","Data":"7cb8b70d3cb09158c17d0d728b88dce459d4560cd25f79902c00aa5c2995d388"} Oct 30 22:04:02.768556 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:02.768322 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cb8b70d3cb09158c17d0d728b88dce459d4560cd25f79902c00aa5c2995d388" Oct 30 22:04:03.572690 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:03.572658 4335 patch_prober.go:29] interesting pod/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:04:03.572948 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:03.572922 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" podUID="c4940530-3248-4c16-b8b0-5959219c8aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 22:04:04.571632 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:04.571580 4335 patch_prober.go:29] interesting pod/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:04:04.572017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:04.571660 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" podUID="c4940530-3248-4c16-b8b0-5959219c8aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 22:04:04.841542 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:04.841408 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn"] Oct 30 22:04:04.842114 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:04.842061 4335 cpu_manager.go:414] "containerMap: removing container" podUID="c82efd63-6d58-424c-bcff-772354c678f9" containerName="revoke-keys" Oct 30 22:04:04.842114 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:04.842081 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c82efd63-6d58-424c-bcff-772354c678f9" containerName="revoke-keys" Oct 30 22:04:04.842242 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:04.842199 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="c82efd63-6d58-424c-bcff-772354c678f9" containerName="revoke-keys" Oct 30 22:04:04.846174 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:04.846155 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn" Oct 30 22:04:04.862920 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:04.862894 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn"] Oct 30 22:04:04.876635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:04.876275 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b2d9a7-fc83-452d-a515-3e579dd3b79f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ec3c884-7ggzn\" (UID: \"34b2d9a7-fc83-452d-a515-3e579dd3b79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn" Oct 30 22:04:04.876635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:04.876344 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmgw7\" (UniqueName: \"kubernetes.io/projected/34b2d9a7-fc83-452d-a515-3e579dd3b79f-kube-api-access-gmgw7\") pod \"maas-api-revoke-sub-3ec3c884-7ggzn\" (UID: \"34b2d9a7-fc83-452d-a515-3e579dd3b79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn" Oct 30 22:04:04.876635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:04.876445 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b2d9a7-fc83-452d-a515-3e579dd3b79f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ec3c884-7ggzn\" (UID: \"34b2d9a7-fc83-452d-a515-3e579dd3b79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn" Oct 30 22:04:04.904866 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:04.904843 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:04:04.905057 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:04.905042 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:04.909785 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:04.909765 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:04.910001 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:04.909986 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:04.998081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:04.998046 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b2d9a7-fc83-452d-a515-3e579dd3b79f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ec3c884-7ggzn\" (UID: \"34b2d9a7-fc83-452d-a515-3e579dd3b79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn" Oct 30 22:04:05.004693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:05.004654 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmgw7\" (UniqueName: \"kubernetes.io/projected/34b2d9a7-fc83-452d-a515-3e579dd3b79f-kube-api-access-gmgw7\") pod \"maas-api-revoke-sub-3ec3c884-7ggzn\" (UID: \"34b2d9a7-fc83-452d-a515-3e579dd3b79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn" Oct 30 22:04:05.048638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:05.041847 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b2d9a7-fc83-452d-a515-3e579dd3b79f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ec3c884-7ggzn\" (UID: \"34b2d9a7-fc83-452d-a515-3e579dd3b79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn" Oct 30 22:04:05.048638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:05.048132 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b2d9a7-fc83-452d-a515-3e579dd3b79f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ec3c884-7ggzn\" (UID: \"34b2d9a7-fc83-452d-a515-3e579dd3b79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn" Oct 30 22:04:05.048638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:05.031124 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b2d9a7-fc83-452d-a515-3e579dd3b79f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ec3c884-7ggzn\" (UID: \"34b2d9a7-fc83-452d-a515-3e579dd3b79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn" Oct 30 22:04:05.067514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:05.067483 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmgw7\" (UniqueName: \"kubernetes.io/projected/34b2d9a7-fc83-452d-a515-3e579dd3b79f-kube-api-access-gmgw7\") pod \"maas-api-revoke-sub-3ec3c884-7ggzn\" (UID: \"34b2d9a7-fc83-452d-a515-3e579dd3b79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn" Oct 30 22:04:05.164538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:05.164452 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn" Oct 30 22:04:05.572317 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:05.572026 4335 patch_prober.go:29] interesting pod/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:04:05.572317 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:05.572093 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" podUID="c4940530-3248-4c16-b8b0-5959219c8aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 22:04:05.726071 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:05.725798 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn"] Oct 30 22:04:05.726211 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:04:05.726079 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b2d9a7_fc83_452d_a515_3e579dd3b79f.slice/crio-fcf22b9b1184ea8d914f5426221396ae516d58c45382d4c1aa5a4a2b48fcec75 WatchSource:0}: Error finding container fcf22b9b1184ea8d914f5426221396ae516d58c45382d4c1aa5a4a2b48fcec75: Status 404 returned error can't find the container with id fcf22b9b1184ea8d914f5426221396ae516d58c45382d4c1aa5a4a2b48fcec75 Oct 30 22:04:05.846744 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:05.846648 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn" event={"ID":"34b2d9a7-fc83-452d-a515-3e579dd3b79f","Type":"ContainerStarted","Data":"fcf22b9b1184ea8d914f5426221396ae516d58c45382d4c1aa5a4a2b48fcec75"} Oct 30 22:04:06.571664 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:06.571605 4335 patch_prober.go:29] interesting pod/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:04:06.571824 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:06.571677 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" podUID="c4940530-3248-4c16-b8b0-5959219c8aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 22:04:06.859897 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:06.859866 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn" event={"ID":"34b2d9a7-fc83-452d-a515-3e579dd3b79f","Type":"ContainerStarted","Data":"a511ad002a3edf5f56c68c4a1054db33f7da6e215759026dd947a2562cce5f79"} Oct 30 22:04:06.983474 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:06.983426 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn" podStartSLOduration=2.983412011 podStartE2EDuration="2.983412011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:06.980364154 +0000 UTC m=+1180.080795887" watchObservedRunningTime="2026-10-30 22:04:06.983412011 +0000 UTC m=+1180.083843763" Oct 30 22:04:07.571978 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:07.571941 4335 patch_prober.go:29] interesting pod/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:04:07.572174 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:07.572031 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" podUID="c4940530-3248-4c16-b8b0-5959219c8aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 22:04:07.869150 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:07.869054 4335 generic.go:356] "Generic (PLEG): container finished" podID="34b2d9a7-fc83-452d-a515-3e579dd3b79f" containerID="a511ad002a3edf5f56c68c4a1054db33f7da6e215759026dd947a2562cce5f79" exitCode=0 Oct 30 22:04:07.869150 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:07.869139 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn" event={"ID":"34b2d9a7-fc83-452d-a515-3e579dd3b79f","Type":"ContainerDied","Data":"a511ad002a3edf5f56c68c4a1054db33f7da6e215759026dd947a2562cce5f79"} Oct 30 22:04:08.579895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:08.579864 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:08.580351 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:08.580330 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:08.580898 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:08.580874 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:04:08.803250 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:08.803223 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6"] Oct 30 22:04:08.807310 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:08.807294 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6" Oct 30 22:04:08.865591 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:08.865501 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6"] Oct 30 22:04:09.020733 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.020680 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf71281e-6d19-47db-9e68-cd63215f0824-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94d3ee89-p5jg6\" (UID: \"cf71281e-6d19-47db-9e68-cd63215f0824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6" Oct 30 22:04:09.021197 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.020758 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls9dk\" (UniqueName: \"kubernetes.io/projected/cf71281e-6d19-47db-9e68-cd63215f0824-kube-api-access-ls9dk\") pod \"maas-api-revoke-sub-94d3ee89-p5jg6\" (UID: \"cf71281e-6d19-47db-9e68-cd63215f0824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6" Oct 30 22:04:09.021197 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.020906 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf71281e-6d19-47db-9e68-cd63215f0824-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94d3ee89-p5jg6\" (UID: \"cf71281e-6d19-47db-9e68-cd63215f0824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6" Oct 30 22:04:09.103024 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.102995 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn" Oct 30 22:04:09.122355 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.122287 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf71281e-6d19-47db-9e68-cd63215f0824-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94d3ee89-p5jg6\" (UID: \"cf71281e-6d19-47db-9e68-cd63215f0824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6" Oct 30 22:04:09.122479 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.122467 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf71281e-6d19-47db-9e68-cd63215f0824-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94d3ee89-p5jg6\" (UID: \"cf71281e-6d19-47db-9e68-cd63215f0824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6" Oct 30 22:04:09.122552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.122504 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls9dk\" (UniqueName: \"kubernetes.io/projected/cf71281e-6d19-47db-9e68-cd63215f0824-kube-api-access-ls9dk\") pod \"maas-api-revoke-sub-94d3ee89-p5jg6\" (UID: \"cf71281e-6d19-47db-9e68-cd63215f0824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6" Oct 30 22:04:09.123033 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.123002 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf71281e-6d19-47db-9e68-cd63215f0824-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94d3ee89-p5jg6\" (UID: \"cf71281e-6d19-47db-9e68-cd63215f0824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6" Oct 30 22:04:09.125031 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.125004 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf71281e-6d19-47db-9e68-cd63215f0824-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94d3ee89-p5jg6\" (UID: \"cf71281e-6d19-47db-9e68-cd63215f0824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6" Oct 30 22:04:09.183529 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.183501 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls9dk\" (UniqueName: \"kubernetes.io/projected/cf71281e-6d19-47db-9e68-cd63215f0824-kube-api-access-ls9dk\") pod \"maas-api-revoke-sub-94d3ee89-p5jg6\" (UID: \"cf71281e-6d19-47db-9e68-cd63215f0824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6" Oct 30 22:04:09.223288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.223258 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b2d9a7-fc83-452d-a515-3e579dd3b79f-kube-api-access-gmgw7\" (UniqueName: \"kubernetes.io/projected/34b2d9a7-fc83-452d-a515-3e579dd3b79f-kube-api-access-gmgw7\") pod \"34b2d9a7-fc83-452d-a515-3e579dd3b79f\" (UID: \"34b2d9a7-fc83-452d-a515-3e579dd3b79f\") " Oct 30 22:04:09.223463 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.223445 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34b2d9a7-fc83-452d-a515-3e579dd3b79f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b2d9a7-fc83-452d-a515-3e579dd3b79f-maas-api-xks-ca\") pod \"34b2d9a7-fc83-452d-a515-3e579dd3b79f\" (UID: \"34b2d9a7-fc83-452d-a515-3e579dd3b79f\") " Oct 30 22:04:09.223539 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.223494 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b2d9a7-fc83-452d-a515-3e579dd3b79f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b2d9a7-fc83-452d-a515-3e579dd3b79f-maas-api-openshift-ca\") pod \"34b2d9a7-fc83-452d-a515-3e579dd3b79f\" (UID: \"34b2d9a7-fc83-452d-a515-3e579dd3b79f\") " Oct 30 22:04:09.224086 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.224052 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b2d9a7-fc83-452d-a515-3e579dd3b79f-maas-api-openshift-ca" pod "34b2d9a7-fc83-452d-a515-3e579dd3b79f" (UID: "34b2d9a7-fc83-452d-a515-3e579dd3b79f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:09.225941 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.225913 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b2d9a7-fc83-452d-a515-3e579dd3b79f-maas-api-xks-ca" pod "34b2d9a7-fc83-452d-a515-3e579dd3b79f" (UID: "34b2d9a7-fc83-452d-a515-3e579dd3b79f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:09.226315 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.226300 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b2d9a7-fc83-452d-a515-3e579dd3b79f-kube-api-access-gmgw7" pod "34b2d9a7-fc83-452d-a515-3e579dd3b79f" (UID: "34b2d9a7-fc83-452d-a515-3e579dd3b79f"). InnerVolumeSpecName "kube-api-access-gmgw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:09.325111 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.325071 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b2d9a7-fc83-452d-a515-3e579dd3b79f-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:09.325111 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.325109 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b2d9a7-fc83-452d-a515-3e579dd3b79f-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:09.325286 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.325128 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmgw7\" (UniqueName: \"kubernetes.io/projected/34b2d9a7-fc83-452d-a515-3e579dd3b79f-kube-api-access-gmgw7\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:09.422360 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.422284 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6" Oct 30 22:04:09.711805 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.711715 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv"] Oct 30 22:04:09.712668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.712649 4335 cpu_manager.go:414] "containerMap: removing container" podUID="34b2d9a7-fc83-452d-a515-3e579dd3b79f" containerName="revoke-keys" Oct 30 22:04:09.712668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.712667 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b2d9a7-fc83-452d-a515-3e579dd3b79f" containerName="revoke-keys" Oct 30 22:04:09.712862 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.712825 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b2d9a7-fc83-452d-a515-3e579dd3b79f" containerName="revoke-keys" Oct 30 22:04:09.717478 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.717458 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv" Oct 30 22:04:09.770445 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.770415 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv"] Oct 30 22:04:09.806397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.806370 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6"] Oct 30 22:04:09.832461 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.832428 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a54b8c68-7xlbv\" (UID: \"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv" Oct 30 22:04:09.832689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.832478 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8jxm\" (UniqueName: \"kubernetes.io/projected/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-kube-api-access-p8jxm\") pod \"maas-api-revoke-sub-a54b8c68-7xlbv\" (UID: \"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv" Oct 30 22:04:09.832689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.832563 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a54b8c68-7xlbv\" (UID: \"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv" Oct 30 22:04:09.893341 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.893312 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6" event={"ID":"cf71281e-6d19-47db-9e68-cd63215f0824","Type":"ContainerStarted","Data":"dee77629c2a12ce70da2461753a7b49a88c4512350070476e334a8e52ee13450"} Oct 30 22:04:09.894915 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.894889 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn" Oct 30 22:04:09.894915 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.894905 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn" event={"ID":"34b2d9a7-fc83-452d-a515-3e579dd3b79f","Type":"ContainerDied","Data":"fcf22b9b1184ea8d914f5426221396ae516d58c45382d4c1aa5a4a2b48fcec75"} Oct 30 22:04:09.895073 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.894953 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcf22b9b1184ea8d914f5426221396ae516d58c45382d4c1aa5a4a2b48fcec75" Oct 30 22:04:09.934055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.934031 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a54b8c68-7xlbv\" (UID: \"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv" Oct 30 22:04:09.934163 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.934069 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8jxm\" (UniqueName: \"kubernetes.io/projected/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-kube-api-access-p8jxm\") pod \"maas-api-revoke-sub-a54b8c68-7xlbv\" (UID: \"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv" Oct 30 22:04:09.934163 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.934154 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a54b8c68-7xlbv\" (UID: \"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv" Oct 30 22:04:09.934724 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.934702 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a54b8c68-7xlbv\" (UID: \"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv" Oct 30 22:04:09.936672 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.936654 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a54b8c68-7xlbv\" (UID: \"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv" Oct 30 22:04:09.995209 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:09.995162 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8jxm\" (UniqueName: \"kubernetes.io/projected/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-kube-api-access-p8jxm\") pod \"maas-api-revoke-sub-a54b8c68-7xlbv\" (UID: \"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv" Oct 30 22:04:10.031757 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:10.031731 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv" Oct 30 22:04:10.418842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:10.418813 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv"] Oct 30 22:04:10.419402 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:04:10.419377 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab13cda_bcde_45d7_bb98_ee11ddcdfc49.slice/crio-e319d4db80c287257ba0285a8815298b55bb06631908a75f46469dfc65a78734 WatchSource:0}: Error finding container e319d4db80c287257ba0285a8815298b55bb06631908a75f46469dfc65a78734: Status 404 returned error can't find the container with id e319d4db80c287257ba0285a8815298b55bb06631908a75f46469dfc65a78734 Oct 30 22:04:10.903817 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:10.903787 4335 generic.go:356] "Generic (PLEG): container finished" podID="cf71281e-6d19-47db-9e68-cd63215f0824" containerID="0a221fa58522dd429a6c5072e1e77343374c2ad3a3d9daf3353c85da914b57a7" exitCode=0 Oct 30 22:04:10.903969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:10.903866 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6" event={"ID":"cf71281e-6d19-47db-9e68-cd63215f0824","Type":"ContainerDied","Data":"0a221fa58522dd429a6c5072e1e77343374c2ad3a3d9daf3353c85da914b57a7"} Oct 30 22:04:10.905552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:10.905530 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv" event={"ID":"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49","Type":"ContainerStarted","Data":"bf31fae34bf69693d1b88b4875a039475d7a8decf00363d688e6e4c37cdf85ea"} Oct 30 22:04:10.905699 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:10.905569 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv" event={"ID":"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49","Type":"ContainerStarted","Data":"e319d4db80c287257ba0285a8815298b55bb06631908a75f46469dfc65a78734"} Oct 30 22:04:11.135286 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:11.135228 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv" podStartSLOduration=2.135209785 podStartE2EDuration="2.135209785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:11.133433042 +0000 UTC m=+1184.233864774" watchObservedRunningTime="2026-10-30 22:04:11.135209785 +0000 UTC m=+1184.235641518" Oct 30 22:04:11.915187 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:11.915152 4335 generic.go:356] "Generic (PLEG): container finished" podID="9ab13cda-bcde-45d7-bb98-ee11ddcdfc49" containerID="bf31fae34bf69693d1b88b4875a039475d7a8decf00363d688e6e4c37cdf85ea" exitCode=0 Oct 30 22:04:11.915341 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:11.915238 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv" event={"ID":"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49","Type":"ContainerDied","Data":"bf31fae34bf69693d1b88b4875a039475d7a8decf00363d688e6e4c37cdf85ea"} Oct 30 22:04:12.065868 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:12.065844 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6" Oct 30 22:04:12.259107 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:12.259037 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf71281e-6d19-47db-9e68-cd63215f0824-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf71281e-6d19-47db-9e68-cd63215f0824-maas-api-openshift-ca\") pod \"cf71281e-6d19-47db-9e68-cd63215f0824\" (UID: \"cf71281e-6d19-47db-9e68-cd63215f0824\") " Oct 30 22:04:12.259107 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:12.259102 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf71281e-6d19-47db-9e68-cd63215f0824-kube-api-access-ls9dk\" (UniqueName: \"kubernetes.io/projected/cf71281e-6d19-47db-9e68-cd63215f0824-kube-api-access-ls9dk\") pod \"cf71281e-6d19-47db-9e68-cd63215f0824\" (UID: \"cf71281e-6d19-47db-9e68-cd63215f0824\") " Oct 30 22:04:12.259521 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:12.259179 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf71281e-6d19-47db-9e68-cd63215f0824-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf71281e-6d19-47db-9e68-cd63215f0824-maas-api-xks-ca\") pod \"cf71281e-6d19-47db-9e68-cd63215f0824\" (UID: \"cf71281e-6d19-47db-9e68-cd63215f0824\") " Oct 30 22:04:12.259521 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:12.259410 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf71281e-6d19-47db-9e68-cd63215f0824-maas-api-openshift-ca" pod "cf71281e-6d19-47db-9e68-cd63215f0824" (UID: "cf71281e-6d19-47db-9e68-cd63215f0824"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:12.259723 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:12.259704 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf71281e-6d19-47db-9e68-cd63215f0824-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:12.261521 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:12.261496 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf71281e-6d19-47db-9e68-cd63215f0824-maas-api-xks-ca" pod "cf71281e-6d19-47db-9e68-cd63215f0824" (UID: "cf71281e-6d19-47db-9e68-cd63215f0824"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:12.261521 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:12.261508 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf71281e-6d19-47db-9e68-cd63215f0824-kube-api-access-ls9dk" pod "cf71281e-6d19-47db-9e68-cd63215f0824" (UID: "cf71281e-6d19-47db-9e68-cd63215f0824"). InnerVolumeSpecName "kube-api-access-ls9dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:12.360961 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:12.360927 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls9dk\" (UniqueName: \"kubernetes.io/projected/cf71281e-6d19-47db-9e68-cd63215f0824-kube-api-access-ls9dk\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:12.360961 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:12.360958 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf71281e-6d19-47db-9e68-cd63215f0824-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:12.925558 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:12.925527 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6" Oct 30 22:04:12.925746 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:12.925559 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6" event={"ID":"cf71281e-6d19-47db-9e68-cd63215f0824","Type":"ContainerDied","Data":"dee77629c2a12ce70da2461753a7b49a88c4512350070476e334a8e52ee13450"} Oct 30 22:04:12.925746 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:12.925631 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dee77629c2a12ce70da2461753a7b49a88c4512350070476e334a8e52ee13450" Oct 30 22:04:13.109057 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:13.109031 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv" Oct 30 22:04:13.169031 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:13.168999 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-maas-api-openshift-ca\") pod \"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49\" (UID: \"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49\") " Oct 30 22:04:13.169173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:13.169076 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-kube-api-access-p8jxm\" (UniqueName: \"kubernetes.io/projected/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-kube-api-access-p8jxm\") pod \"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49\" (UID: \"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49\") " Oct 30 22:04:13.169173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:13.169107 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-maas-api-xks-ca\") pod \"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49\" (UID: \"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49\") " Oct 30 22:04:13.169419 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:13.169393 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-maas-api-openshift-ca" pod "9ab13cda-bcde-45d7-bb98-ee11ddcdfc49" (UID: "9ab13cda-bcde-45d7-bb98-ee11ddcdfc49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:13.169630 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:13.169593 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:13.171324 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:13.171298 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-maas-api-xks-ca" pod "9ab13cda-bcde-45d7-bb98-ee11ddcdfc49" (UID: "9ab13cda-bcde-45d7-bb98-ee11ddcdfc49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:13.171394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:13.171376 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-kube-api-access-p8jxm" pod "9ab13cda-bcde-45d7-bb98-ee11ddcdfc49" (UID: "9ab13cda-bcde-45d7-bb98-ee11ddcdfc49"). InnerVolumeSpecName "kube-api-access-p8jxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:13.270329 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:13.270298 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8jxm\" (UniqueName: \"kubernetes.io/projected/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-kube-api-access-p8jxm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:13.270329 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:13.270324 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:13.935478 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:13.935452 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv" Oct 30 22:04:13.935663 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:13.935492 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv" event={"ID":"9ab13cda-bcde-45d7-bb98-ee11ddcdfc49","Type":"ContainerDied","Data":"e319d4db80c287257ba0285a8815298b55bb06631908a75f46469dfc65a78734"} Oct 30 22:04:13.935663 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:13.935540 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e319d4db80c287257ba0285a8815298b55bb06631908a75f46469dfc65a78734" Oct 30 22:04:18.036120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.036086 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247"] Oct 30 22:04:18.036724 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.036705 4335 cpu_manager.go:414] "containerMap: removing container" podUID="9ab13cda-bcde-45d7-bb98-ee11ddcdfc49" containerName="revoke-keys" Oct 30 22:04:18.036819 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.036727 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab13cda-bcde-45d7-bb98-ee11ddcdfc49" containerName="revoke-keys" Oct 30 22:04:18.036819 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.036748 4335 cpu_manager.go:414] "containerMap: removing container" podUID="cf71281e-6d19-47db-9e68-cd63215f0824" containerName="revoke-keys" Oct 30 22:04:18.036819 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.036757 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf71281e-6d19-47db-9e68-cd63215f0824" containerName="revoke-keys" Oct 30 22:04:18.036975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.036962 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab13cda-bcde-45d7-bb98-ee11ddcdfc49" containerName="revoke-keys" Oct 30 22:04:18.037017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.036979 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf71281e-6d19-47db-9e68-cd63215f0824" containerName="revoke-keys" Oct 30 22:04:18.040323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.040307 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247" Oct 30 22:04:18.100120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.100092 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247"] Oct 30 22:04:18.118724 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.118687 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tsvm\" (UniqueName: \"kubernetes.io/projected/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-kube-api-access-6tsvm\") pod \"maas-api-revoke-sub-89834086-m5247\" (UID: \"9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247" Oct 30 22:04:18.118867 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.118856 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89834086-m5247\" (UID: \"9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247" Oct 30 22:04:18.118949 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.118903 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89834086-m5247\" (UID: \"9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247" Oct 30 22:04:18.142499 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.142477 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:18.142602 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.142481 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:04:18.142602 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.142521 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:18.142602 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.142568 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:18.220296 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.220270 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89834086-m5247\" (UID: \"9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247" Oct 30 22:04:18.220430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.220339 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89834086-m5247\" (UID: \"9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247" Oct 30 22:04:18.220430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.220373 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tsvm\" (UniqueName: \"kubernetes.io/projected/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-kube-api-access-6tsvm\") pod \"maas-api-revoke-sub-89834086-m5247\" (UID: \"9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247" Oct 30 22:04:18.220972 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.220939 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89834086-m5247\" (UID: \"9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247" Oct 30 22:04:18.222769 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.222740 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89834086-m5247\" (UID: \"9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247" Oct 30 22:04:18.281071 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.281045 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tsvm\" (UniqueName: \"kubernetes.io/projected/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-kube-api-access-6tsvm\") pod \"maas-api-revoke-sub-89834086-m5247\" (UID: \"9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247" Oct 30 22:04:18.355265 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.355242 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247" Oct 30 22:04:18.740285 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:04:18.740251 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3145bc_4a50_4b9e_bfa1_e81ed0cac3f9.slice/crio-df1a61b98dc300a929d5435487af2e1a500bda2ba53eddd900d3a0179f0a55d3 WatchSource:0}: Error finding container df1a61b98dc300a929d5435487af2e1a500bda2ba53eddd900d3a0179f0a55d3: Status 404 returned error can't find the container with id df1a61b98dc300a929d5435487af2e1a500bda2ba53eddd900d3a0179f0a55d3 Oct 30 22:04:18.742444 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.742419 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247"] Oct 30 22:04:18.976135 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:18.976106 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247" event={"ID":"9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9","Type":"ContainerStarted","Data":"df1a61b98dc300a929d5435487af2e1a500bda2ba53eddd900d3a0179f0a55d3"} Oct 30 22:04:19.985772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:19.985690 4335 generic.go:356] "Generic (PLEG): container finished" podID="9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9" containerID="db1ed4f9fa1d644cb2bb9d35c0ab821bf7444f6d9f8b6edc77b31d57d9347a8f" exitCode=0 Oct 30 22:04:19.985772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:19.985735 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247" event={"ID":"9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9","Type":"ContainerDied","Data":"db1ed4f9fa1d644cb2bb9d35c0ab821bf7444f6d9f8b6edc77b31d57d9347a8f"} Oct 30 22:04:21.152874 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.152851 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247" Oct 30 22:04:21.253332 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.253266 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-maas-api-openshift-ca\") pod \"9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9\" (UID: \"9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9\") " Oct 30 22:04:21.253332 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.253305 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-kube-api-access-6tsvm\" (UniqueName: \"kubernetes.io/projected/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-kube-api-access-6tsvm\") pod \"9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9\" (UID: \"9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9\") " Oct 30 22:04:21.253509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.253344 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-maas-api-xks-ca\") pod \"9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9\" (UID: \"9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9\") " Oct 30 22:04:21.253689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.253665 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-maas-api-openshift-ca" pod "9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9" (UID: "9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:21.255546 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.255517 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-maas-api-xks-ca" pod "9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9" (UID: "9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:21.255741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.255717 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-kube-api-access-6tsvm" pod "9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9" (UID: "9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9"). InnerVolumeSpecName "kube-api-access-6tsvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:21.354609 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.354574 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:21.354753 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.354629 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tsvm\" (UniqueName: \"kubernetes.io/projected/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-kube-api-access-6tsvm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:21.354753 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.354645 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:21.561427 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.561397 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq"] Oct 30 22:04:21.562053 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.562032 4335 cpu_manager.go:414] "containerMap: removing container" podUID="9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9" containerName="revoke-keys" Oct 30 22:04:21.562053 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.562053 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9" containerName="revoke-keys" Oct 30 22:04:21.562226 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.562213 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9" containerName="revoke-keys" Oct 30 22:04:21.565704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.565683 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq" Oct 30 22:04:21.622382 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.622357 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq"] Oct 30 22:04:21.658196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.658160 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61841cc6-12d1-4821-bc3e-fe1efd7e6763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1f01cf7-rfbcq\" (UID: \"61841cc6-12d1-4821-bc3e-fe1efd7e6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq" Oct 30 22:04:21.658349 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.658214 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmthm\" (UniqueName: \"kubernetes.io/projected/61841cc6-12d1-4821-bc3e-fe1efd7e6763-kube-api-access-jmthm\") pod \"maas-api-revoke-sub-d1f01cf7-rfbcq\" (UID: \"61841cc6-12d1-4821-bc3e-fe1efd7e6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq" Oct 30 22:04:21.658477 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.658452 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61841cc6-12d1-4821-bc3e-fe1efd7e6763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1f01cf7-rfbcq\" (UID: \"61841cc6-12d1-4821-bc3e-fe1efd7e6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq" Oct 30 22:04:21.678040 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.677990 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl"] Oct 30 22:04:21.682298 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.682276 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl" Oct 30 22:04:21.738344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.738321 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl"] Oct 30 22:04:21.759712 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.759684 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2964de57-6bd4-472c-b74a-4a60b17d7a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f43fbfe-zbkzl\" (UID: \"2964de57-6bd4-472c-b74a-4a60b17d7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl" Oct 30 22:04:21.759839 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.759754 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61841cc6-12d1-4821-bc3e-fe1efd7e6763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1f01cf7-rfbcq\" (UID: \"61841cc6-12d1-4821-bc3e-fe1efd7e6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq" Oct 30 22:04:21.759839 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.759782 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv8vd\" (UniqueName: \"kubernetes.io/projected/2964de57-6bd4-472c-b74a-4a60b17d7a08-kube-api-access-lv8vd\") pod \"maas-api-revoke-sub-9f43fbfe-zbkzl\" (UID: \"2964de57-6bd4-472c-b74a-4a60b17d7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl" Oct 30 22:04:21.759932 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.759876 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2964de57-6bd4-472c-b74a-4a60b17d7a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f43fbfe-zbkzl\" (UID: \"2964de57-6bd4-472c-b74a-4a60b17d7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl" Oct 30 22:04:21.759932 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.759897 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61841cc6-12d1-4821-bc3e-fe1efd7e6763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1f01cf7-rfbcq\" (UID: \"61841cc6-12d1-4821-bc3e-fe1efd7e6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq" Oct 30 22:04:21.760056 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.760033 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmthm\" (UniqueName: \"kubernetes.io/projected/61841cc6-12d1-4821-bc3e-fe1efd7e6763-kube-api-access-jmthm\") pod \"maas-api-revoke-sub-d1f01cf7-rfbcq\" (UID: \"61841cc6-12d1-4821-bc3e-fe1efd7e6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq" Oct 30 22:04:21.760501 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.760435 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61841cc6-12d1-4821-bc3e-fe1efd7e6763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1f01cf7-rfbcq\" (UID: \"61841cc6-12d1-4821-bc3e-fe1efd7e6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq" Oct 30 22:04:21.762384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.762360 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61841cc6-12d1-4821-bc3e-fe1efd7e6763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1f01cf7-rfbcq\" (UID: \"61841cc6-12d1-4821-bc3e-fe1efd7e6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq" Oct 30 22:04:21.822134 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.822063 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmthm\" (UniqueName: \"kubernetes.io/projected/61841cc6-12d1-4821-bc3e-fe1efd7e6763-kube-api-access-jmthm\") pod \"maas-api-revoke-sub-d1f01cf7-rfbcq\" (UID: \"61841cc6-12d1-4821-bc3e-fe1efd7e6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq" Oct 30 22:04:21.861310 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.861284 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2964de57-6bd4-472c-b74a-4a60b17d7a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f43fbfe-zbkzl\" (UID: \"2964de57-6bd4-472c-b74a-4a60b17d7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl" Oct 30 22:04:21.861461 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.861444 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2964de57-6bd4-472c-b74a-4a60b17d7a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f43fbfe-zbkzl\" (UID: \"2964de57-6bd4-472c-b74a-4a60b17d7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl" Oct 30 22:04:21.861517 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.861497 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv8vd\" (UniqueName: \"kubernetes.io/projected/2964de57-6bd4-472c-b74a-4a60b17d7a08-kube-api-access-lv8vd\") pod \"maas-api-revoke-sub-9f43fbfe-zbkzl\" (UID: \"2964de57-6bd4-472c-b74a-4a60b17d7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl" Oct 30 22:04:21.862055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.862034 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2964de57-6bd4-472c-b74a-4a60b17d7a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f43fbfe-zbkzl\" (UID: \"2964de57-6bd4-472c-b74a-4a60b17d7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl" Oct 30 22:04:21.863776 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.863752 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2964de57-6bd4-472c-b74a-4a60b17d7a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f43fbfe-zbkzl\" (UID: \"2964de57-6bd4-472c-b74a-4a60b17d7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl" Oct 30 22:04:21.880667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.880646 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq" Oct 30 22:04:21.924260 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.924229 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv8vd\" (UniqueName: \"kubernetes.io/projected/2964de57-6bd4-472c-b74a-4a60b17d7a08-kube-api-access-lv8vd\") pod \"maas-api-revoke-sub-9f43fbfe-zbkzl\" (UID: \"2964de57-6bd4-472c-b74a-4a60b17d7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl" Oct 30 22:04:21.996932 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:21.996903 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl" Oct 30 22:04:22.005024 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:22.004997 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247" event={"ID":"9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9","Type":"ContainerDied","Data":"df1a61b98dc300a929d5435487af2e1a500bda2ba53eddd900d3a0179f0a55d3"} Oct 30 22:04:22.005132 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:22.005039 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247" Oct 30 22:04:22.005132 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:22.005048 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df1a61b98dc300a929d5435487af2e1a500bda2ba53eddd900d3a0179f0a55d3" Oct 30 22:04:22.263532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:22.263505 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq"] Oct 30 22:04:22.264194 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:04:22.264151 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61841cc6_12d1_4821_bc3e_fe1efd7e6763.slice/crio-9afcce10f3aad015b10e0851b83a6fe485f5f811df339e2b0eb275b784281974 WatchSource:0}: Error finding container 9afcce10f3aad015b10e0851b83a6fe485f5f811df339e2b0eb275b784281974: Status 404 returned error can't find the container with id 9afcce10f3aad015b10e0851b83a6fe485f5f811df339e2b0eb275b784281974 Oct 30 22:04:22.394416 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:22.394374 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl"] Oct 30 22:04:22.395016 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:04:22.394984 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2964de57_6bd4_472c_b74a_4a60b17d7a08.slice/crio-8e0ab0c732419ef0cc59704510f522e333bd1ea2480c43876069816a68432c3e WatchSource:0}: Error finding container 8e0ab0c732419ef0cc59704510f522e333bd1ea2480c43876069816a68432c3e: Status 404 returned error can't find the container with id 8e0ab0c732419ef0cc59704510f522e333bd1ea2480c43876069816a68432c3e Oct 30 22:04:23.017005 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:23.016969 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl" event={"ID":"2964de57-6bd4-472c-b74a-4a60b17d7a08","Type":"ContainerStarted","Data":"538bfd9ccef295a1709afa55728622bf11c257d83649e6639c7879139fd39938"} Oct 30 22:04:23.017141 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:23.017067 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl" event={"ID":"2964de57-6bd4-472c-b74a-4a60b17d7a08","Type":"ContainerStarted","Data":"8e0ab0c732419ef0cc59704510f522e333bd1ea2480c43876069816a68432c3e"} Oct 30 22:04:23.022746 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:23.022675 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq" event={"ID":"61841cc6-12d1-4821-bc3e-fe1efd7e6763","Type":"ContainerStarted","Data":"9741a25ef70beb6bf8f5cd1b2067b63e4a7040478604963e01b3dfdd4bdfd08d"} Oct 30 22:04:23.022746 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:23.022737 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq" event={"ID":"61841cc6-12d1-4821-bc3e-fe1efd7e6763","Type":"ContainerStarted","Data":"9afcce10f3aad015b10e0851b83a6fe485f5f811df339e2b0eb275b784281974"} Oct 30 22:04:23.138747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:23.138660 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl" podStartSLOduration=2.138643558 podStartE2EDuration="2.138643558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:23.131438265 +0000 UTC m=+1196.231869996" watchObservedRunningTime="2026-10-30 22:04:23.138643558 +0000 UTC m=+1196.239075292" Oct 30 22:04:23.250061 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:23.250014 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq" podStartSLOduration=2.250000913 podStartE2EDuration="2.250000913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:23.245928686 +0000 UTC m=+1196.346360420" watchObservedRunningTime="2026-10-30 22:04:23.250000913 +0000 UTC m=+1196.350432692" Oct 30 22:04:24.034351 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:24.034317 4335 generic.go:356] "Generic (PLEG): container finished" podID="2964de57-6bd4-472c-b74a-4a60b17d7a08" containerID="538bfd9ccef295a1709afa55728622bf11c257d83649e6639c7879139fd39938" exitCode=0 Oct 30 22:04:24.034793 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:24.034483 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl" event={"ID":"2964de57-6bd4-472c-b74a-4a60b17d7a08","Type":"ContainerDied","Data":"538bfd9ccef295a1709afa55728622bf11c257d83649e6639c7879139fd39938"} Oct 30 22:04:24.036642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:24.036598 4335 generic.go:356] "Generic (PLEG): container finished" podID="61841cc6-12d1-4821-bc3e-fe1efd7e6763" containerID="9741a25ef70beb6bf8f5cd1b2067b63e4a7040478604963e01b3dfdd4bdfd08d" exitCode=0 Oct 30 22:04:24.036771 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:24.036712 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq" event={"ID":"61841cc6-12d1-4821-bc3e-fe1efd7e6763","Type":"ContainerDied","Data":"9741a25ef70beb6bf8f5cd1b2067b63e4a7040478604963e01b3dfdd4bdfd08d"} Oct 30 22:04:25.218653 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.216569 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl" Oct 30 22:04:25.256418 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.256392 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq" Oct 30 22:04:25.300080 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.300058 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2964de57-6bd4-472c-b74a-4a60b17d7a08-kube-api-access-lv8vd\" (UniqueName: \"kubernetes.io/projected/2964de57-6bd4-472c-b74a-4a60b17d7a08-kube-api-access-lv8vd\") pod \"2964de57-6bd4-472c-b74a-4a60b17d7a08\" (UID: \"2964de57-6bd4-472c-b74a-4a60b17d7a08\") " Oct 30 22:04:25.300229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.300149 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2964de57-6bd4-472c-b74a-4a60b17d7a08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2964de57-6bd4-472c-b74a-4a60b17d7a08-maas-api-xks-ca\") pod \"2964de57-6bd4-472c-b74a-4a60b17d7a08\" (UID: \"2964de57-6bd4-472c-b74a-4a60b17d7a08\") " Oct 30 22:04:25.300297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.300275 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2964de57-6bd4-472c-b74a-4a60b17d7a08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2964de57-6bd4-472c-b74a-4a60b17d7a08-maas-api-openshift-ca\") pod \"2964de57-6bd4-472c-b74a-4a60b17d7a08\" (UID: \"2964de57-6bd4-472c-b74a-4a60b17d7a08\") " Oct 30 22:04:25.300348 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.300330 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61841cc6-12d1-4821-bc3e-fe1efd7e6763-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61841cc6-12d1-4821-bc3e-fe1efd7e6763-maas-api-openshift-ca\") pod \"61841cc6-12d1-4821-bc3e-fe1efd7e6763\" (UID: \"61841cc6-12d1-4821-bc3e-fe1efd7e6763\") " Oct 30 22:04:25.300430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.300416 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61841cc6-12d1-4821-bc3e-fe1efd7e6763-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61841cc6-12d1-4821-bc3e-fe1efd7e6763-maas-api-xks-ca\") pod \"61841cc6-12d1-4821-bc3e-fe1efd7e6763\" (UID: \"61841cc6-12d1-4821-bc3e-fe1efd7e6763\") " Oct 30 22:04:25.300487 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.300473 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61841cc6-12d1-4821-bc3e-fe1efd7e6763-kube-api-access-jmthm\" (UniqueName: \"kubernetes.io/projected/61841cc6-12d1-4821-bc3e-fe1efd7e6763-kube-api-access-jmthm\") pod \"61841cc6-12d1-4821-bc3e-fe1efd7e6763\" (UID: \"61841cc6-12d1-4821-bc3e-fe1efd7e6763\") " Oct 30 22:04:25.301455 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.301402 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2964de57-6bd4-472c-b74a-4a60b17d7a08-maas-api-openshift-ca" pod "2964de57-6bd4-472c-b74a-4a60b17d7a08" (UID: "2964de57-6bd4-472c-b74a-4a60b17d7a08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:25.301586 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.301560 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61841cc6-12d1-4821-bc3e-fe1efd7e6763-maas-api-openshift-ca" pod "61841cc6-12d1-4821-bc3e-fe1efd7e6763" (UID: "61841cc6-12d1-4821-bc3e-fe1efd7e6763"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:25.303056 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.302642 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2964de57-6bd4-472c-b74a-4a60b17d7a08-kube-api-access-lv8vd" pod "2964de57-6bd4-472c-b74a-4a60b17d7a08" (UID: "2964de57-6bd4-472c-b74a-4a60b17d7a08"). InnerVolumeSpecName "kube-api-access-lv8vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:25.303056 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.302850 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2964de57-6bd4-472c-b74a-4a60b17d7a08-maas-api-xks-ca" pod "2964de57-6bd4-472c-b74a-4a60b17d7a08" (UID: "2964de57-6bd4-472c-b74a-4a60b17d7a08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:25.303056 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.303375 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61841cc6-12d1-4821-bc3e-fe1efd7e6763-kube-api-access-jmthm" pod "61841cc6-12d1-4821-bc3e-fe1efd7e6763" (UID: "61841cc6-12d1-4821-bc3e-fe1efd7e6763"). InnerVolumeSpecName "kube-api-access-jmthm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:25.305839 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.305814 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61841cc6-12d1-4821-bc3e-fe1efd7e6763-maas-api-xks-ca" pod "61841cc6-12d1-4821-bc3e-fe1efd7e6763" (UID: "61841cc6-12d1-4821-bc3e-fe1efd7e6763"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:25.326836 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:04:25.326809 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:04:25.402156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.402127 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2964de57-6bd4-472c-b74a-4a60b17d7a08-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:25.402287 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.402161 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2964de57-6bd4-472c-b74a-4a60b17d7a08-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:25.402287 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.402176 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61841cc6-12d1-4821-bc3e-fe1efd7e6763-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:25.402287 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.402192 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61841cc6-12d1-4821-bc3e-fe1efd7e6763-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:25.402287 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.402207 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmthm\" (UniqueName: \"kubernetes.io/projected/61841cc6-12d1-4821-bc3e-fe1efd7e6763-kube-api-access-jmthm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:25.402287 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.402221 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv8vd\" (UniqueName: \"kubernetes.io/projected/2964de57-6bd4-472c-b74a-4a60b17d7a08-kube-api-access-lv8vd\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:25.964594 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.964565 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp"] Oct 30 22:04:25.965404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.965382 4335 cpu_manager.go:414] "containerMap: removing container" podUID="61841cc6-12d1-4821-bc3e-fe1efd7e6763" containerName="revoke-keys" Oct 30 22:04:25.965404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.965402 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61841cc6-12d1-4821-bc3e-fe1efd7e6763" containerName="revoke-keys" Oct 30 22:04:25.965584 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.965426 4335 cpu_manager.go:414] "containerMap: removing container" podUID="2964de57-6bd4-472c-b74a-4a60b17d7a08" containerName="revoke-keys" Oct 30 22:04:25.965584 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.965434 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2964de57-6bd4-472c-b74a-4a60b17d7a08" containerName="revoke-keys" Oct 30 22:04:25.965584 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.965548 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="61841cc6-12d1-4821-bc3e-fe1efd7e6763" containerName="revoke-keys" Oct 30 22:04:25.965584 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.965563 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="2964de57-6bd4-472c-b74a-4a60b17d7a08" containerName="revoke-keys" Oct 30 22:04:25.969852 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.969831 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp" Oct 30 22:04:25.989578 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:25.989556 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:04:26.008436 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:26.008402 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc4f728e-5rmrp\" (UID: \"8f4a2fcd-7972-4f11-a57c-17c8f9c46f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp" Oct 30 22:04:26.008565 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:26.008540 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc4f728e-5rmrp\" (UID: \"8f4a2fcd-7972-4f11-a57c-17c8f9c46f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp" Oct 30 22:04:26.008760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:26.008739 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crhpg\" (UniqueName: \"kubernetes.io/projected/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-kube-api-access-crhpg\") pod \"maas-api-revoke-sub-cc4f728e-5rmrp\" (UID: \"8f4a2fcd-7972-4f11-a57c-17c8f9c46f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp" Oct 30 22:04:26.026492 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:26.026469 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp"] Oct 30 22:04:26.056488 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:26.056450 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl" event={"ID":"2964de57-6bd4-472c-b74a-4a60b17d7a08","Type":"ContainerDied","Data":"8e0ab0c732419ef0cc59704510f522e333bd1ea2480c43876069816a68432c3e"} Oct 30 22:04:26.056661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:26.056495 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl" Oct 30 22:04:26.056661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:26.056509 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e0ab0c732419ef0cc59704510f522e333bd1ea2480c43876069816a68432c3e" Oct 30 22:04:26.058270 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:26.058188 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq" Oct 30 22:04:26.058270 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:26.058226 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq" event={"ID":"61841cc6-12d1-4821-bc3e-fe1efd7e6763","Type":"ContainerDied","Data":"9afcce10f3aad015b10e0851b83a6fe485f5f811df339e2b0eb275b784281974"} Oct 30 22:04:26.058448 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:26.058270 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9afcce10f3aad015b10e0851b83a6fe485f5f811df339e2b0eb275b784281974" Oct 30 22:04:26.109463 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:26.109430 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crhpg\" (UniqueName: \"kubernetes.io/projected/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-kube-api-access-crhpg\") pod \"maas-api-revoke-sub-cc4f728e-5rmrp\" (UID: \"8f4a2fcd-7972-4f11-a57c-17c8f9c46f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp" Oct 30 22:04:26.109595 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:26.109548 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc4f728e-5rmrp\" (UID: \"8f4a2fcd-7972-4f11-a57c-17c8f9c46f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp" Oct 30 22:04:26.109978 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:26.109695 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc4f728e-5rmrp\" (UID: \"8f4a2fcd-7972-4f11-a57c-17c8f9c46f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp" Oct 30 22:04:26.110527 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:26.110508 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc4f728e-5rmrp\" (UID: \"8f4a2fcd-7972-4f11-a57c-17c8f9c46f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp" Oct 30 22:04:26.112546 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:26.112526 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc4f728e-5rmrp\" (UID: \"8f4a2fcd-7972-4f11-a57c-17c8f9c46f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp" Oct 30 22:04:26.189388 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:26.189348 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crhpg\" (UniqueName: \"kubernetes.io/projected/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-kube-api-access-crhpg\") pod \"maas-api-revoke-sub-cc4f728e-5rmrp\" (UID: \"8f4a2fcd-7972-4f11-a57c-17c8f9c46f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp" Oct 30 22:04:26.292716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:26.292630 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp" Oct 30 22:04:26.713646 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:26.712324 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp"] Oct 30 22:04:27.384074 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:27.384038 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp" event={"ID":"8f4a2fcd-7972-4f11-a57c-17c8f9c46f51","Type":"ContainerStarted","Data":"949cc2baa0a51268b4dba7b75d90a373120cbe6778cb041edb4c77e688a30a6d"} Oct 30 22:04:27.644640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:27.623806 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:04:27.644640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:27.636570 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:04:28.393437 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:28.393405 4335 generic.go:356] "Generic (PLEG): container finished" podID="8f4a2fcd-7972-4f11-a57c-17c8f9c46f51" containerID="60641eadffad410c29faf7694b68be1d7b50a193507c0e3cff103bfc92def616" exitCode=0 Oct 30 22:04:28.393847 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:28.393498 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp" event={"ID":"8f4a2fcd-7972-4f11-a57c-17c8f9c46f51","Type":"ContainerDied","Data":"60641eadffad410c29faf7694b68be1d7b50a193507c0e3cff103bfc92def616"} Oct 30 22:04:29.565115 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:29.565089 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp" Oct 30 22:04:29.651281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:29.651256 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-maas-api-openshift-ca\") pod \"8f4a2fcd-7972-4f11-a57c-17c8f9c46f51\" (UID: \"8f4a2fcd-7972-4f11-a57c-17c8f9c46f51\") " Oct 30 22:04:29.651391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:29.651340 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-maas-api-xks-ca\") pod \"8f4a2fcd-7972-4f11-a57c-17c8f9c46f51\" (UID: \"8f4a2fcd-7972-4f11-a57c-17c8f9c46f51\") " Oct 30 22:04:29.651440 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:29.651425 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-kube-api-access-crhpg\" (UniqueName: \"kubernetes.io/projected/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-kube-api-access-crhpg\") pod \"8f4a2fcd-7972-4f11-a57c-17c8f9c46f51\" (UID: \"8f4a2fcd-7972-4f11-a57c-17c8f9c46f51\") " Oct 30 22:04:29.651579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:29.651558 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-maas-api-openshift-ca" pod "8f4a2fcd-7972-4f11-a57c-17c8f9c46f51" (UID: "8f4a2fcd-7972-4f11-a57c-17c8f9c46f51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:29.651853 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:29.651836 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:29.653537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:29.653490 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-maas-api-xks-ca" pod "8f4a2fcd-7972-4f11-a57c-17c8f9c46f51" (UID: "8f4a2fcd-7972-4f11-a57c-17c8f9c46f51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:29.653630 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:29.653582 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-kube-api-access-crhpg" pod "8f4a2fcd-7972-4f11-a57c-17c8f9c46f51" (UID: "8f4a2fcd-7972-4f11-a57c-17c8f9c46f51"). InnerVolumeSpecName "kube-api-access-crhpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:29.752779 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:29.752749 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crhpg\" (UniqueName: \"kubernetes.io/projected/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-kube-api-access-crhpg\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:29.752779 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:29.752776 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:30.412606 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:30.412578 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp" Oct 30 22:04:30.412829 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:30.412628 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp" event={"ID":"8f4a2fcd-7972-4f11-a57c-17c8f9c46f51","Type":"ContainerDied","Data":"949cc2baa0a51268b4dba7b75d90a373120cbe6778cb041edb4c77e688a30a6d"} Oct 30 22:04:30.412829 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:30.412692 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="949cc2baa0a51268b4dba7b75d90a373120cbe6778cb041edb4c77e688a30a6d" Oct 30 22:04:33.979865 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:33.979832 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh"] Oct 30 22:04:33.980591 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:33.980571 4335 cpu_manager.go:414] "containerMap: removing container" podUID="8f4a2fcd-7972-4f11-a57c-17c8f9c46f51" containerName="revoke-keys" Oct 30 22:04:33.980746 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:33.980592 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f4a2fcd-7972-4f11-a57c-17c8f9c46f51" containerName="revoke-keys" Oct 30 22:04:33.980807 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:33.980798 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f4a2fcd-7972-4f11-a57c-17c8f9c46f51" containerName="revoke-keys" Oct 30 22:04:33.985368 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:33.985346 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh" Oct 30 22:04:34.037391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:34.037368 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:34.086710 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:34.086681 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:34.086835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:34.086743 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:04:34.086835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:34.086781 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:34.091376 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:34.091358 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh"] Oct 30 22:04:34.095308 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:34.095289 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf66bbd9-61d9-432c-9e5b-a525eb60945a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0c1780d-c8xsh\" (UID: \"bf66bbd9-61d9-432c-9e5b-a525eb60945a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh" Oct 30 22:04:34.095391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:34.095373 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf66bbd9-61d9-432c-9e5b-a525eb60945a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0c1780d-c8xsh\" (UID: \"bf66bbd9-61d9-432c-9e5b-a525eb60945a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh" Oct 30 22:04:34.095512 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:34.095495 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpn4h\" (UniqueName: \"kubernetes.io/projected/bf66bbd9-61d9-432c-9e5b-a525eb60945a-kube-api-access-dpn4h\") pod \"maas-api-revoke-sub-c0c1780d-c8xsh\" (UID: \"bf66bbd9-61d9-432c-9e5b-a525eb60945a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh" Oct 30 22:04:34.196234 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:34.196208 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf66bbd9-61d9-432c-9e5b-a525eb60945a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0c1780d-c8xsh\" (UID: \"bf66bbd9-61d9-432c-9e5b-a525eb60945a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh" Oct 30 22:04:34.196355 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:34.196269 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf66bbd9-61d9-432c-9e5b-a525eb60945a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0c1780d-c8xsh\" (UID: \"bf66bbd9-61d9-432c-9e5b-a525eb60945a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh" Oct 30 22:04:34.196422 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:34.196368 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpn4h\" (UniqueName: \"kubernetes.io/projected/bf66bbd9-61d9-432c-9e5b-a525eb60945a-kube-api-access-dpn4h\") pod \"maas-api-revoke-sub-c0c1780d-c8xsh\" (UID: \"bf66bbd9-61d9-432c-9e5b-a525eb60945a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh" Oct 30 22:04:34.196926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:34.196903 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf66bbd9-61d9-432c-9e5b-a525eb60945a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0c1780d-c8xsh\" (UID: \"bf66bbd9-61d9-432c-9e5b-a525eb60945a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh" Oct 30 22:04:34.198808 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:34.198784 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf66bbd9-61d9-432c-9e5b-a525eb60945a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0c1780d-c8xsh\" (UID: \"bf66bbd9-61d9-432c-9e5b-a525eb60945a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh" Oct 30 22:04:34.257563 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:34.257512 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpn4h\" (UniqueName: \"kubernetes.io/projected/bf66bbd9-61d9-432c-9e5b-a525eb60945a-kube-api-access-dpn4h\") pod \"maas-api-revoke-sub-c0c1780d-c8xsh\" (UID: \"bf66bbd9-61d9-432c-9e5b-a525eb60945a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh" Oct 30 22:04:34.298721 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:34.298700 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh" Oct 30 22:04:34.677633 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:04:34.677586 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf66bbd9_61d9_432c_9e5b_a525eb60945a.slice/crio-ca5f33bc5cb24d3f23e43ad559590f190d70d9bb5a12187eabfe9f46f2e55f9b WatchSource:0}: Error finding container ca5f33bc5cb24d3f23e43ad559590f190d70d9bb5a12187eabfe9f46f2e55f9b: Status 404 returned error can't find the container with id ca5f33bc5cb24d3f23e43ad559590f190d70d9bb5a12187eabfe9f46f2e55f9b Oct 30 22:04:34.678210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:34.678153 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh"] Oct 30 22:04:35.455238 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:35.455209 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh" event={"ID":"bf66bbd9-61d9-432c-9e5b-a525eb60945a","Type":"ContainerStarted","Data":"2cad413f5c27cf55e67d179d9516b4b91e5c8e94cf9f4aa899bbe59970326c33"} Oct 30 22:04:35.455713 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:35.455256 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh" event={"ID":"bf66bbd9-61d9-432c-9e5b-a525eb60945a","Type":"ContainerStarted","Data":"ca5f33bc5cb24d3f23e43ad559590f190d70d9bb5a12187eabfe9f46f2e55f9b"} Oct 30 22:04:35.572835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:35.572784 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh" podStartSLOduration=2.572770656 podStartE2EDuration="2.572770656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:35.569919596 +0000 UTC m=+1208.670351329" watchObservedRunningTime="2026-10-30 22:04:35.572770656 +0000 UTC m=+1208.673202388" Oct 30 22:04:36.464143 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:36.464113 4335 generic.go:356] "Generic (PLEG): container finished" podID="bf66bbd9-61d9-432c-9e5b-a525eb60945a" containerID="2cad413f5c27cf55e67d179d9516b4b91e5c8e94cf9f4aa899bbe59970326c33" exitCode=0 Oct 30 22:04:36.464544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:36.464199 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh" event={"ID":"bf66bbd9-61d9-432c-9e5b-a525eb60945a","Type":"ContainerDied","Data":"2cad413f5c27cf55e67d179d9516b4b91e5c8e94cf9f4aa899bbe59970326c33"} Oct 30 22:04:36.723176 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:36.723107 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d"] Oct 30 22:04:36.727159 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:36.727141 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d" Oct 30 22:04:36.785215 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:36.785176 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d"] Oct 30 22:04:36.824776 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:36.824754 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf5pg\" (UniqueName: \"kubernetes.io/projected/7b4b181a-794b-4f14-8091-f0e821da1bdb-kube-api-access-mf5pg\") pod \"maas-api-revoke-sub-5c2be64e-7vj4d\" (UID: \"7b4b181a-794b-4f14-8091-f0e821da1bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d" Oct 30 22:04:36.824910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:36.824800 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4b181a-794b-4f14-8091-f0e821da1bdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c2be64e-7vj4d\" (UID: \"7b4b181a-794b-4f14-8091-f0e821da1bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d" Oct 30 22:04:36.824958 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:36.824938 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4b181a-794b-4f14-8091-f0e821da1bdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c2be64e-7vj4d\" (UID: \"7b4b181a-794b-4f14-8091-f0e821da1bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d" Oct 30 22:04:36.925824 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:36.925792 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf5pg\" (UniqueName: \"kubernetes.io/projected/7b4b181a-794b-4f14-8091-f0e821da1bdb-kube-api-access-mf5pg\") pod \"maas-api-revoke-sub-5c2be64e-7vj4d\" (UID: \"7b4b181a-794b-4f14-8091-f0e821da1bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d" Oct 30 22:04:36.925995 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:36.925838 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4b181a-794b-4f14-8091-f0e821da1bdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c2be64e-7vj4d\" (UID: \"7b4b181a-794b-4f14-8091-f0e821da1bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d" Oct 30 22:04:36.925995 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:36.925885 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4b181a-794b-4f14-8091-f0e821da1bdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c2be64e-7vj4d\" (UID: \"7b4b181a-794b-4f14-8091-f0e821da1bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d" Oct 30 22:04:36.926531 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:36.926506 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4b181a-794b-4f14-8091-f0e821da1bdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c2be64e-7vj4d\" (UID: \"7b4b181a-794b-4f14-8091-f0e821da1bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d" Oct 30 22:04:36.928203 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:36.928180 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4b181a-794b-4f14-8091-f0e821da1bdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c2be64e-7vj4d\" (UID: \"7b4b181a-794b-4f14-8091-f0e821da1bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d" Oct 30 22:04:36.986526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:36.986463 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf5pg\" (UniqueName: \"kubernetes.io/projected/7b4b181a-794b-4f14-8091-f0e821da1bdb-kube-api-access-mf5pg\") pod \"maas-api-revoke-sub-5c2be64e-7vj4d\" (UID: \"7b4b181a-794b-4f14-8091-f0e821da1bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d" Oct 30 22:04:37.040746 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:37.040719 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d" Oct 30 22:04:37.424996 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:04:37.424967 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4b181a_794b_4f14_8091_f0e821da1bdb.slice/crio-c1d08d931baec7b2f9dac459f7a6415cf1a4b9e122f4ae4ede477bac84bd9d36 WatchSource:0}: Error finding container c1d08d931baec7b2f9dac459f7a6415cf1a4b9e122f4ae4ede477bac84bd9d36: Status 404 returned error can't find the container with id c1d08d931baec7b2f9dac459f7a6415cf1a4b9e122f4ae4ede477bac84bd9d36 Oct 30 22:04:37.425968 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:37.425880 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d"] Oct 30 22:04:37.473934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:37.473908 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d" event={"ID":"7b4b181a-794b-4f14-8091-f0e821da1bdb","Type":"ContainerStarted","Data":"c1d08d931baec7b2f9dac459f7a6415cf1a4b9e122f4ae4ede477bac84bd9d36"} Oct 30 22:04:37.618029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:37.618004 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh" Oct 30 22:04:37.734457 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:37.734431 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf66bbd9-61d9-432c-9e5b-a525eb60945a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf66bbd9-61d9-432c-9e5b-a525eb60945a-maas-api-xks-ca\") pod \"bf66bbd9-61d9-432c-9e5b-a525eb60945a\" (UID: \"bf66bbd9-61d9-432c-9e5b-a525eb60945a\") " Oct 30 22:04:37.734595 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:37.734559 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf66bbd9-61d9-432c-9e5b-a525eb60945a-kube-api-access-dpn4h\" (UniqueName: \"kubernetes.io/projected/bf66bbd9-61d9-432c-9e5b-a525eb60945a-kube-api-access-dpn4h\") pod \"bf66bbd9-61d9-432c-9e5b-a525eb60945a\" (UID: \"bf66bbd9-61d9-432c-9e5b-a525eb60945a\") " Oct 30 22:04:37.734725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:37.734687 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf66bbd9-61d9-432c-9e5b-a525eb60945a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf66bbd9-61d9-432c-9e5b-a525eb60945a-maas-api-openshift-ca\") pod \"bf66bbd9-61d9-432c-9e5b-a525eb60945a\" (UID: \"bf66bbd9-61d9-432c-9e5b-a525eb60945a\") " Oct 30 22:04:37.736196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:37.736138 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf66bbd9-61d9-432c-9e5b-a525eb60945a-maas-api-openshift-ca" pod "bf66bbd9-61d9-432c-9e5b-a525eb60945a" (UID: "bf66bbd9-61d9-432c-9e5b-a525eb60945a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:37.737423 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:37.737397 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf66bbd9-61d9-432c-9e5b-a525eb60945a-maas-api-xks-ca" pod "bf66bbd9-61d9-432c-9e5b-a525eb60945a" (UID: "bf66bbd9-61d9-432c-9e5b-a525eb60945a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:37.737889 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:37.737815 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf66bbd9-61d9-432c-9e5b-a525eb60945a-kube-api-access-dpn4h" pod "bf66bbd9-61d9-432c-9e5b-a525eb60945a" (UID: "bf66bbd9-61d9-432c-9e5b-a525eb60945a"). InnerVolumeSpecName "kube-api-access-dpn4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:37.837266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:37.837235 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf66bbd9-61d9-432c-9e5b-a525eb60945a-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:37.837266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:37.837261 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf66bbd9-61d9-432c-9e5b-a525eb60945a-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:37.837266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:37.837271 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpn4h\" (UniqueName: \"kubernetes.io/projected/bf66bbd9-61d9-432c-9e5b-a525eb60945a-kube-api-access-dpn4h\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:38.482593 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:38.482510 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh" Oct 30 22:04:38.482593 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:38.482539 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh" event={"ID":"bf66bbd9-61d9-432c-9e5b-a525eb60945a","Type":"ContainerDied","Data":"ca5f33bc5cb24d3f23e43ad559590f190d70d9bb5a12187eabfe9f46f2e55f9b"} Oct 30 22:04:38.482593 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:38.482594 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca5f33bc5cb24d3f23e43ad559590f190d70d9bb5a12187eabfe9f46f2e55f9b" Oct 30 22:04:38.484523 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:38.484499 4335 generic.go:356] "Generic (PLEG): container finished" podID="7b4b181a-794b-4f14-8091-f0e821da1bdb" containerID="76b558689b717d9340ce40d63ff336c71164d03aa12752fa43c4834fbe74cfda" exitCode=0 Oct 30 22:04:38.484664 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:38.484567 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d" event={"ID":"7b4b181a-794b-4f14-8091-f0e821da1bdb","Type":"ContainerDied","Data":"76b558689b717d9340ce40d63ff336c71164d03aa12752fa43c4834fbe74cfda"} Oct 30 22:04:39.649438 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:39.649418 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d" Oct 30 22:04:39.760098 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:39.760039 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4b181a-794b-4f14-8091-f0e821da1bdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4b181a-794b-4f14-8091-f0e821da1bdb-maas-api-openshift-ca\") pod \"7b4b181a-794b-4f14-8091-f0e821da1bdb\" (UID: \"7b4b181a-794b-4f14-8091-f0e821da1bdb\") " Oct 30 22:04:39.760209 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:39.760185 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b4b181a-794b-4f14-8091-f0e821da1bdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4b181a-794b-4f14-8091-f0e821da1bdb-maas-api-xks-ca\") pod \"7b4b181a-794b-4f14-8091-f0e821da1bdb\" (UID: \"7b4b181a-794b-4f14-8091-f0e821da1bdb\") " Oct 30 22:04:39.760249 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:39.760213 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4b181a-794b-4f14-8091-f0e821da1bdb-kube-api-access-mf5pg\" (UniqueName: \"kubernetes.io/projected/7b4b181a-794b-4f14-8091-f0e821da1bdb-kube-api-access-mf5pg\") pod \"7b4b181a-794b-4f14-8091-f0e821da1bdb\" (UID: \"7b4b181a-794b-4f14-8091-f0e821da1bdb\") " Oct 30 22:04:39.760359 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:39.760337 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4b181a-794b-4f14-8091-f0e821da1bdb-maas-api-openshift-ca" pod "7b4b181a-794b-4f14-8091-f0e821da1bdb" (UID: "7b4b181a-794b-4f14-8091-f0e821da1bdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:39.760687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:39.760663 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4b181a-794b-4f14-8091-f0e821da1bdb-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:39.762418 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:39.762388 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b4b181a-794b-4f14-8091-f0e821da1bdb-maas-api-xks-ca" pod "7b4b181a-794b-4f14-8091-f0e821da1bdb" (UID: "7b4b181a-794b-4f14-8091-f0e821da1bdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:39.762504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:39.762411 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4b181a-794b-4f14-8091-f0e821da1bdb-kube-api-access-mf5pg" pod "7b4b181a-794b-4f14-8091-f0e821da1bdb" (UID: "7b4b181a-794b-4f14-8091-f0e821da1bdb"). InnerVolumeSpecName "kube-api-access-mf5pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:39.862185 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:39.862153 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4b181a-794b-4f14-8091-f0e821da1bdb-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:39.862185 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:39.862187 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf5pg\" (UniqueName: \"kubernetes.io/projected/7b4b181a-794b-4f14-8091-f0e821da1bdb-kube-api-access-mf5pg\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:40.502697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:40.502661 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d" event={"ID":"7b4b181a-794b-4f14-8091-f0e821da1bdb","Type":"ContainerDied","Data":"c1d08d931baec7b2f9dac459f7a6415cf1a4b9e122f4ae4ede477bac84bd9d36"} Oct 30 22:04:40.502887 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:40.502709 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d" Oct 30 22:04:40.502887 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:40.502713 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1d08d931baec7b2f9dac459f7a6415cf1a4b9e122f4ae4ede477bac84bd9d36" Oct 30 22:04:52.631641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.631598 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p"] Oct 30 22:04:52.634146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.632208 4335 cpu_manager.go:414] "containerMap: removing container" podUID="7b4b181a-794b-4f14-8091-f0e821da1bdb" containerName="revoke-keys" Oct 30 22:04:52.634146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.632220 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4b181a-794b-4f14-8091-f0e821da1bdb" containerName="revoke-keys" Oct 30 22:04:52.634146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.632247 4335 cpu_manager.go:414] "containerMap: removing container" podUID="bf66bbd9-61d9-432c-9e5b-a525eb60945a" containerName="revoke-keys" Oct 30 22:04:52.634146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.632252 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf66bbd9-61d9-432c-9e5b-a525eb60945a" containerName="revoke-keys" Oct 30 22:04:52.634146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.632362 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf66bbd9-61d9-432c-9e5b-a525eb60945a" containerName="revoke-keys" Oct 30 22:04:52.634146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.632376 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4b181a-794b-4f14-8091-f0e821da1bdb" containerName="revoke-keys" Oct 30 22:04:52.635750 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.635735 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p" Oct 30 22:04:52.688450 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.688431 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:52.688641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.688484 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:52.688943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.688923 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:52.738296 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.738254 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:04:52.741131 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.741102 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27824a71-6977-42de-afbb-c3578649bf70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f274bbf6-7xk2p\" (UID: \"27824a71-6977-42de-afbb-c3578649bf70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p" Oct 30 22:04:52.741286 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.741184 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npt7w\" (UniqueName: \"kubernetes.io/projected/27824a71-6977-42de-afbb-c3578649bf70-kube-api-access-npt7w\") pod \"maas-api-revoke-sub-f274bbf6-7xk2p\" (UID: \"27824a71-6977-42de-afbb-c3578649bf70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p" Oct 30 22:04:52.741456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.741428 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27824a71-6977-42de-afbb-c3578649bf70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f274bbf6-7xk2p\" (UID: \"27824a71-6977-42de-afbb-c3578649bf70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p" Oct 30 22:04:52.743715 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.743692 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p"] Oct 30 22:04:52.842558 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.842534 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27824a71-6977-42de-afbb-c3578649bf70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f274bbf6-7xk2p\" (UID: \"27824a71-6977-42de-afbb-c3578649bf70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p" Oct 30 22:04:52.842673 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.842578 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27824a71-6977-42de-afbb-c3578649bf70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f274bbf6-7xk2p\" (UID: \"27824a71-6977-42de-afbb-c3578649bf70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p" Oct 30 22:04:52.842673 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.842651 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npt7w\" (UniqueName: \"kubernetes.io/projected/27824a71-6977-42de-afbb-c3578649bf70-kube-api-access-npt7w\") pod \"maas-api-revoke-sub-f274bbf6-7xk2p\" (UID: \"27824a71-6977-42de-afbb-c3578649bf70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p" Oct 30 22:04:52.843196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.843173 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27824a71-6977-42de-afbb-c3578649bf70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f274bbf6-7xk2p\" (UID: \"27824a71-6977-42de-afbb-c3578649bf70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p" Oct 30 22:04:52.845077 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.845050 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27824a71-6977-42de-afbb-c3578649bf70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f274bbf6-7xk2p\" (UID: \"27824a71-6977-42de-afbb-c3578649bf70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p" Oct 30 22:04:52.903842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.903793 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npt7w\" (UniqueName: \"kubernetes.io/projected/27824a71-6977-42de-afbb-c3578649bf70-kube-api-access-npt7w\") pod \"maas-api-revoke-sub-f274bbf6-7xk2p\" (UID: \"27824a71-6977-42de-afbb-c3578649bf70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p" Oct 30 22:04:52.948987 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:52.948963 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p" Oct 30 22:04:53.328970 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:04:53.328937 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27824a71_6977_42de_afbb_c3578649bf70.slice/crio-2c68911e4a29e5527191b4c9783a48541e5714403121a736b3cb4134f0b139b6 WatchSource:0}: Error finding container 2c68911e4a29e5527191b4c9783a48541e5714403121a736b3cb4134f0b139b6: Status 404 returned error can't find the container with id 2c68911e4a29e5527191b4c9783a48541e5714403121a736b3cb4134f0b139b6 Oct 30 22:04:53.330392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:53.330364 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p"] Oct 30 22:04:53.618150 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:53.618122 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p" event={"ID":"27824a71-6977-42de-afbb-c3578649bf70","Type":"ContainerStarted","Data":"04c3444def8bce47b441ced27af67b83479e52253f6f5d150e70cb92fecd312c"} Oct 30 22:04:53.618276 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:53.618185 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p" event={"ID":"27824a71-6977-42de-afbb-c3578649bf70","Type":"ContainerStarted","Data":"2c68911e4a29e5527191b4c9783a48541e5714403121a736b3cb4134f0b139b6"} Oct 30 22:04:53.734705 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:53.734661 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p" podStartSLOduration=1.734647887 podStartE2EDuration="1.734647887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:53.732767514 +0000 UTC m=+1226.833199247" watchObservedRunningTime="2026-10-30 22:04:53.734647887 +0000 UTC m=+1226.835079615" Oct 30 22:04:54.627517 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:54.627489 4335 generic.go:356] "Generic (PLEG): container finished" podID="27824a71-6977-42de-afbb-c3578649bf70" containerID="04c3444def8bce47b441ced27af67b83479e52253f6f5d150e70cb92fecd312c" exitCode=0 Oct 30 22:04:54.627723 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:54.627577 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p" event={"ID":"27824a71-6977-42de-afbb-c3578649bf70","Type":"ContainerDied","Data":"04c3444def8bce47b441ced27af67b83479e52253f6f5d150e70cb92fecd312c"} Oct 30 22:04:55.320051 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:04:55.320016 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:04:55.802304 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:55.802276 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p" Oct 30 22:04:55.874668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:55.874642 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27824a71-6977-42de-afbb-c3578649bf70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27824a71-6977-42de-afbb-c3578649bf70-maas-api-openshift-ca\") pod \"27824a71-6977-42de-afbb-c3578649bf70\" (UID: \"27824a71-6977-42de-afbb-c3578649bf70\") " Oct 30 22:04:55.874853 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:55.874717 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27824a71-6977-42de-afbb-c3578649bf70-kube-api-access-npt7w\" (UniqueName: \"kubernetes.io/projected/27824a71-6977-42de-afbb-c3578649bf70-kube-api-access-npt7w\") pod \"27824a71-6977-42de-afbb-c3578649bf70\" (UID: \"27824a71-6977-42de-afbb-c3578649bf70\") " Oct 30 22:04:55.874853 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:55.874775 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27824a71-6977-42de-afbb-c3578649bf70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27824a71-6977-42de-afbb-c3578649bf70-maas-api-xks-ca\") pod \"27824a71-6977-42de-afbb-c3578649bf70\" (UID: \"27824a71-6977-42de-afbb-c3578649bf70\") " Oct 30 22:04:55.874987 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:55.874962 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27824a71-6977-42de-afbb-c3578649bf70-maas-api-openshift-ca" pod "27824a71-6977-42de-afbb-c3578649bf70" (UID: "27824a71-6977-42de-afbb-c3578649bf70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:55.875125 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:55.875107 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27824a71-6977-42de-afbb-c3578649bf70-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:55.876998 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:55.876974 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27824a71-6977-42de-afbb-c3578649bf70-maas-api-xks-ca" pod "27824a71-6977-42de-afbb-c3578649bf70" (UID: "27824a71-6977-42de-afbb-c3578649bf70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:55.877083 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:55.877050 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27824a71-6977-42de-afbb-c3578649bf70-kube-api-access-npt7w" pod "27824a71-6977-42de-afbb-c3578649bf70" (UID: "27824a71-6977-42de-afbb-c3578649bf70"). InnerVolumeSpecName "kube-api-access-npt7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:55.976786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:55.976721 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npt7w\" (UniqueName: \"kubernetes.io/projected/27824a71-6977-42de-afbb-c3578649bf70-kube-api-access-npt7w\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:55.976786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:55.976747 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27824a71-6977-42de-afbb-c3578649bf70-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:56.644932 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:56.644899 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p" event={"ID":"27824a71-6977-42de-afbb-c3578649bf70","Type":"ContainerDied","Data":"2c68911e4a29e5527191b4c9783a48541e5714403121a736b3cb4134f0b139b6"} Oct 30 22:04:56.644932 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:56.644922 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p" Oct 30 22:04:56.645375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:56.644948 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c68911e4a29e5527191b4c9783a48541e5714403121a736b3cb4134f0b139b6" Oct 30 22:04:59.476867 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.476832 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88"] Oct 30 22:04:59.477405 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.477392 4335 cpu_manager.go:414] "containerMap: removing container" podUID="27824a71-6977-42de-afbb-c3578649bf70" containerName="revoke-keys" Oct 30 22:04:59.477453 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.477405 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27824a71-6977-42de-afbb-c3578649bf70" containerName="revoke-keys" Oct 30 22:04:59.477505 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.477496 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="27824a71-6977-42de-afbb-c3578649bf70" containerName="revoke-keys" Oct 30 22:04:59.484808 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.484780 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88" Oct 30 22:04:59.514207 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.514175 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7175d8a3-d4af-4f8c-87eb-35c096f00e77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb66c93f-plx88\" (UID: \"7175d8a3-d4af-4f8c-87eb-35c096f00e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88" Oct 30 22:04:59.514336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.514245 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7175d8a3-d4af-4f8c-87eb-35c096f00e77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb66c93f-plx88\" (UID: \"7175d8a3-d4af-4f8c-87eb-35c096f00e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88" Oct 30 22:04:59.514336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.514318 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6rck\" (UniqueName: \"kubernetes.io/projected/7175d8a3-d4af-4f8c-87eb-35c096f00e77-kube-api-access-g6rck\") pod \"maas-api-revoke-sub-bb66c93f-plx88\" (UID: \"7175d8a3-d4af-4f8c-87eb-35c096f00e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88" Oct 30 22:04:59.538491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.538470 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88"] Oct 30 22:04:59.582118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.582087 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:04:59.582118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.582090 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:59.582319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.582087 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:59.582319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.582163 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:59.615240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.615214 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7175d8a3-d4af-4f8c-87eb-35c096f00e77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb66c93f-plx88\" (UID: \"7175d8a3-d4af-4f8c-87eb-35c096f00e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88" Oct 30 22:04:59.615371 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.615272 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7175d8a3-d4af-4f8c-87eb-35c096f00e77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb66c93f-plx88\" (UID: \"7175d8a3-d4af-4f8c-87eb-35c096f00e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88" Oct 30 22:04:59.615371 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.615303 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rck\" (UniqueName: \"kubernetes.io/projected/7175d8a3-d4af-4f8c-87eb-35c096f00e77-kube-api-access-g6rck\") pod \"maas-api-revoke-sub-bb66c93f-plx88\" (UID: \"7175d8a3-d4af-4f8c-87eb-35c096f00e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88" Oct 30 22:04:59.615912 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.615891 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7175d8a3-d4af-4f8c-87eb-35c096f00e77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb66c93f-plx88\" (UID: \"7175d8a3-d4af-4f8c-87eb-35c096f00e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88" Oct 30 22:04:59.617795 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.617774 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7175d8a3-d4af-4f8c-87eb-35c096f00e77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb66c93f-plx88\" (UID: \"7175d8a3-d4af-4f8c-87eb-35c096f00e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88" Oct 30 22:04:59.676081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.676060 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6rck\" (UniqueName: \"kubernetes.io/projected/7175d8a3-d4af-4f8c-87eb-35c096f00e77-kube-api-access-g6rck\") pod \"maas-api-revoke-sub-bb66c93f-plx88\" (UID: \"7175d8a3-d4af-4f8c-87eb-35c096f00e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88" Oct 30 22:04:59.799286 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:04:59.799269 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88" Oct 30 22:05:00.177351 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:00.177325 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88"] Oct 30 22:05:00.177700 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:05:00.177676 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7175d8a3_d4af_4f8c_87eb_35c096f00e77.slice/crio-f2352a6dee5d8da53b0798f99e7d5c34c79a166febb1dd533d1faca63aabe129 WatchSource:0}: Error finding container f2352a6dee5d8da53b0798f99e7d5c34c79a166febb1dd533d1faca63aabe129: Status 404 returned error can't find the container with id f2352a6dee5d8da53b0798f99e7d5c34c79a166febb1dd533d1faca63aabe129 Oct 30 22:05:00.676643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:00.676600 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88" event={"ID":"7175d8a3-d4af-4f8c-87eb-35c096f00e77","Type":"ContainerStarted","Data":"f1d78ece05784a6800a18f5ae961f6488558a6505dcc937984213ad6595f79ee"} Oct 30 22:05:00.677074 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:00.676666 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88" event={"ID":"7175d8a3-d4af-4f8c-87eb-35c096f00e77","Type":"ContainerStarted","Data":"f2352a6dee5d8da53b0798f99e7d5c34c79a166febb1dd533d1faca63aabe129"} Oct 30 22:05:00.846992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:00.846935 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88" podStartSLOduration=1.846917192 podStartE2EDuration="1.846917192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:00.79279236 +0000 UTC m=+1233.893224124" watchObservedRunningTime="2026-10-30 22:05:00.846917192 +0000 UTC m=+1233.947348927" Oct 30 22:05:01.631734 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.631704 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b"] Oct 30 22:05:01.635809 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.635789 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.686159 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.686131 4335 generic.go:356] "Generic (PLEG): container finished" podID="7175d8a3-d4af-4f8c-87eb-35c096f00e77" containerID="f1d78ece05784a6800a18f5ae961f6488558a6505dcc937984213ad6595f79ee" exitCode=0 Oct 30 22:05:01.686526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.686211 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88" event={"ID":"7175d8a3-d4af-4f8c-87eb-35c096f00e77","Type":"ContainerDied","Data":"f1d78ece05784a6800a18f5ae961f6488558a6505dcc937984213ad6595f79ee"} Oct 30 22:05:01.737088 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.737053 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-home\") pod \"distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.737276 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.737164 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-model-cache\") pod \"distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.737276 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.737241 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb2bc88a-bb73-4e24-9dce-252844253f65-tls-certs\") pod \"distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.737407 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.737368 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngsdr\" (UniqueName: \"kubernetes.io/projected/fb2bc88a-bb73-4e24-9dce-252844253f65-kube-api-access-ngsdr\") pod \"distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.737407 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.737400 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-tmp-dir\") pod \"distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.737524 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.737496 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-dshm\") pod \"distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.738264 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.738228 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-bf7cbf\"/\"distinct-w1-bf7cbf-kserve-self-signed-certs\"" Oct 30 22:05:01.738410 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.738390 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-bf7cbf\"/\"openshift-service-ca.crt\"" Oct 30 22:05:01.738548 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.738530 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-bf7cbf\"/\"default-dockercfg-z2crq\"" Oct 30 22:05:01.738738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.738564 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-bf7cbf\"/\"kube-root-ca.crt\"" Oct 30 22:05:01.746653 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.746632 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b"] Oct 30 22:05:01.838792 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.838753 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-dshm\") pod \"distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.838976 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.838849 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-home\") pod \"distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.838976 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.838882 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-model-cache\") pod \"distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.838976 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.838918 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb2bc88a-bb73-4e24-9dce-252844253f65-tls-certs\") pod \"distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.839150 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.838994 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngsdr\" (UniqueName: \"kubernetes.io/projected/fb2bc88a-bb73-4e24-9dce-252844253f65-kube-api-access-ngsdr\") pod \"distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.839150 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.839015 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-tmp-dir\") pod \"distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.839298 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.839268 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-home\") pod \"distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.839355 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.839317 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-model-cache\") pod \"distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.839549 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.839516 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-tmp-dir\") pod \"distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.841137 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.841119 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-dshm\") pod \"distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.841875 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.841858 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb2bc88a-bb73-4e24-9dce-252844253f65-tls-certs\") pod \"distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.903486 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.903424 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngsdr\" (UniqueName: \"kubernetes.io/projected/fb2bc88a-bb73-4e24-9dce-252844253f65-kube-api-access-ngsdr\") pod \"distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:01.947991 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:01.947962 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:02.370491 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:05:02.370462 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb2bc88a_bb73_4e24_9dce_252844253f65.slice/crio-bc12c24f6d917c52ad91dec2444c9478022982d8c4c97b4bdc9941d8ef85468f WatchSource:0}: Error finding container bc12c24f6d917c52ad91dec2444c9478022982d8c4c97b4bdc9941d8ef85468f: Status 404 returned error can't find the container with id bc12c24f6d917c52ad91dec2444c9478022982d8c4c97b4bdc9941d8ef85468f Oct 30 22:05:02.371181 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:02.371160 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b"] Oct 30 22:05:02.697911 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:02.697821 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" event={"ID":"fb2bc88a-bb73-4e24-9dce-252844253f65","Type":"ContainerStarted","Data":"538a85c8c90c8cb32af30f0605ed250b624501b0e272e35189f8c48c4cbca517"} Oct 30 22:05:02.697911 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:02.697873 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" event={"ID":"fb2bc88a-bb73-4e24-9dce-252844253f65","Type":"ContainerStarted","Data":"bc12c24f6d917c52ad91dec2444c9478022982d8c4c97b4bdc9941d8ef85468f"} Oct 30 22:05:02.817509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:02.817461 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podStartSLOduration=1.81744695 podStartE2EDuration="1.81744695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:02.815421386 +0000 UTC m=+1235.915853117" watchObservedRunningTime="2026-10-30 22:05:02.81744695 +0000 UTC m=+1235.917878681" Oct 30 22:05:02.836003 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:02.835982 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88" Oct 30 22:05:02.949028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:02.948938 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:02.953995 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:02.953965 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7175d8a3-d4af-4f8c-87eb-35c096f00e77-kube-api-access-g6rck\" (UniqueName: \"kubernetes.io/projected/7175d8a3-d4af-4f8c-87eb-35c096f00e77-kube-api-access-g6rck\") pod \"7175d8a3-d4af-4f8c-87eb-35c096f00e77\" (UID: \"7175d8a3-d4af-4f8c-87eb-35c096f00e77\") " Oct 30 22:05:02.954264 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:02.954228 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7175d8a3-d4af-4f8c-87eb-35c096f00e77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7175d8a3-d4af-4f8c-87eb-35c096f00e77-maas-api-xks-ca\") pod \"7175d8a3-d4af-4f8c-87eb-35c096f00e77\" (UID: \"7175d8a3-d4af-4f8c-87eb-35c096f00e77\") " Oct 30 22:05:02.954365 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:02.954327 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7175d8a3-d4af-4f8c-87eb-35c096f00e77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7175d8a3-d4af-4f8c-87eb-35c096f00e77-maas-api-openshift-ca\") pod \"7175d8a3-d4af-4f8c-87eb-35c096f00e77\" (UID: \"7175d8a3-d4af-4f8c-87eb-35c096f00e77\") " Oct 30 22:05:02.960335 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:02.960301 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7175d8a3-d4af-4f8c-87eb-35c096f00e77-maas-api-openshift-ca" pod "7175d8a3-d4af-4f8c-87eb-35c096f00e77" (UID: "7175d8a3-d4af-4f8c-87eb-35c096f00e77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:02.960793 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:02.960763 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7175d8a3-d4af-4f8c-87eb-35c096f00e77-kube-api-access-g6rck" pod "7175d8a3-d4af-4f8c-87eb-35c096f00e77" (UID: "7175d8a3-d4af-4f8c-87eb-35c096f00e77"). InnerVolumeSpecName "kube-api-access-g6rck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:02.961718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:02.961688 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7175d8a3-d4af-4f8c-87eb-35c096f00e77-maas-api-xks-ca" pod "7175d8a3-d4af-4f8c-87eb-35c096f00e77" (UID: "7175d8a3-d4af-4f8c-87eb-35c096f00e77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:03.056277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:03.056242 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7175d8a3-d4af-4f8c-87eb-35c096f00e77-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.056277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:03.056272 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6rck\" (UniqueName: \"kubernetes.io/projected/7175d8a3-d4af-4f8c-87eb-35c096f00e77-kube-api-access-g6rck\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.056277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:03.056286 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7175d8a3-d4af-4f8c-87eb-35c096f00e77-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.707286 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:03.707260 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88" Oct 30 22:05:03.707758 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:03.707299 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88" event={"ID":"7175d8a3-d4af-4f8c-87eb-35c096f00e77","Type":"ContainerDied","Data":"f2352a6dee5d8da53b0798f99e7d5c34c79a166febb1dd533d1faca63aabe129"} Oct 30 22:05:03.707758 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:03.707346 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2352a6dee5d8da53b0798f99e7d5c34c79a166febb1dd533d1faca63aabe129" Oct 30 22:05:06.310683 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:06.310652 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:05:11.948128 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:11.948094 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:11.961770 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:11.961748 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:12.808729 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:12.808694 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:05:25.320819 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:05:25.320782 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:05:30.320443 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.320405 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx"] Oct 30 22:05:30.321219 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.321201 4335 cpu_manager.go:414] "containerMap: removing container" podUID="7175d8a3-d4af-4f8c-87eb-35c096f00e77" containerName="revoke-keys" Oct 30 22:05:30.321267 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.321221 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7175d8a3-d4af-4f8c-87eb-35c096f00e77" containerName="revoke-keys" Oct 30 22:05:30.321336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.321327 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="7175d8a3-d4af-4f8c-87eb-35c096f00e77" containerName="revoke-keys" Oct 30 22:05:30.325108 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.325092 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.366078 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.366048 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-credential-socket\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.366183 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.366099 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-workload-socket\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.366183 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.366116 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-workload-certs\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.366285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.366263 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f770114f-8048-4957-b7cd-704ffe8fafb1-istiod-ca-cert\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.366321 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.366300 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n58vv\" (UniqueName: \"kubernetes.io/projected/f770114f-8048-4957-b7cd-704ffe8fafb1-kube-api-access-n58vv\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.366417 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.366402 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-envoy\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.366453 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.366428 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-data\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.366487 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.366457 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-token\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.366551 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.366537 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-podinfo\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.434237 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.434204 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx"] Oct 30 22:05:30.467545 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.467520 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-credential-socket\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.467690 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.467579 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-workload-socket\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.467690 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.467606 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-workload-certs\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.467807 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.467755 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f770114f-8048-4957-b7cd-704ffe8fafb1-istiod-ca-cert\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.467807 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.467792 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n58vv\" (UniqueName: \"kubernetes.io/projected/f770114f-8048-4957-b7cd-704ffe8fafb1-kube-api-access-n58vv\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.467901 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.467858 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-envoy\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.467901 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.467886 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-data\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.468008 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.467932 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-token\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.468008 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.467965 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-podinfo\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.468124 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.468009 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-credential-socket\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.468124 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.468033 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-workload-certs\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.468124 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.468098 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-workload-socket\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.468723 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.468689 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-data\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.468723 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.468689 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f770114f-8048-4957-b7cd-704ffe8fafb1-istiod-ca-cert\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.471021 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.470998 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-envoy\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.471446 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.471428 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-podinfo\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.528983 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.528963 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-token\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.529076 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.529044 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n58vv\" (UniqueName: \"kubernetes.io/projected/f770114f-8048-4957-b7cd-704ffe8fafb1-kube-api-access-n58vv\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:30.640310 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:30.640238 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:31.080861 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:05:31.080832 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf770114f_8048_4957_b7cd_704ffe8fafb1.slice/crio-e0880ce5f2e241c02c4e30eee55baf4268f3a39f50b1dd8996f01144a2bf408e WatchSource:0}: Error finding container e0880ce5f2e241c02c4e30eee55baf4268f3a39f50b1dd8996f01144a2bf408e: Status 404 returned error can't find the container with id e0880ce5f2e241c02c4e30eee55baf4268f3a39f50b1dd8996f01144a2bf408e Oct 30 22:05:31.081683 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:31.081658 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx"] Oct 30 22:05:31.083260 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:31.083230 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:05:31.083339 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:31.083297 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:05:31.083339 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:31.083328 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:05:31.972310 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:31.972271 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" event={"ID":"f770114f-8048-4957-b7cd-704ffe8fafb1","Type":"ContainerStarted","Data":"fd4534770da95185d279dc90957b4785043b631afb964f2df0b3f1237745a208"} Oct 30 22:05:31.972865 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:31.972351 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" event={"ID":"f770114f-8048-4957-b7cd-704ffe8fafb1","Type":"ContainerStarted","Data":"e0880ce5f2e241c02c4e30eee55baf4268f3a39f50b1dd8996f01144a2bf408e"} Oct 30 22:05:32.194380 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:32.194328 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" podStartSLOduration=2.194313779 podStartE2EDuration="2.194313779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:32.143080352 +0000 UTC m=+1265.243512095" watchObservedRunningTime="2026-10-30 22:05:32.194313779 +0000 UTC m=+1265.294745510" Oct 30 22:05:32.640589 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:32.640555 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:32.642242 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:32.642217 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:05:32.642387 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:32.642287 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" podUID="f770114f-8048-4957-b7cd-704ffe8fafb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 22:05:33.641228 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:33.641193 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:05:33.641609 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:33.641260 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" podUID="f770114f-8048-4957-b7cd-704ffe8fafb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 22:05:34.641177 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:34.641141 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:05:34.641358 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:34.641212 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" podUID="f770114f-8048-4957-b7cd-704ffe8fafb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 22:05:35.641435 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:35.641403 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:05:35.641875 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:35.641471 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" podUID="f770114f-8048-4957-b7cd-704ffe8fafb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 22:05:36.641464 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:36.641420 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:05:36.642005 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:36.641472 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" podUID="f770114f-8048-4957-b7cd-704ffe8fafb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 22:05:37.793081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:37.793051 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:37.794287 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:37.794266 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:37.795412 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:37.795392 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:05:39.715080 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:39.715045 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:05:41.298034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.297990 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78"] Oct 30 22:05:41.303821 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.303792 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.410473 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.410443 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78"] Oct 30 22:05:41.436574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.436545 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-podinfo\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.436748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.436583 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-workload-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.436748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.436635 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-workload-certs\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.436877 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.436829 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5ldd\" (UniqueName: \"kubernetes.io/projected/d6978eea-f24f-4d30-83b0-85b0015baf53-kube-api-access-d5ldd\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.436953 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.436939 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6978eea-f24f-4d30-83b0-85b0015baf53-istiod-ca-cert\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.436993 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.436974 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-envoy\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.437041 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.437016 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-token\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.437118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.437079 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-data\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.437156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.437137 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-credential-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.537960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.537931 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6978eea-f24f-4d30-83b0-85b0015baf53-istiod-ca-cert\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.538089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.537967 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-envoy\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.538089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.537997 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-token\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.538089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.538016 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-data\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.538089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.538046 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-credential-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.538089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.538084 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-podinfo\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.538426 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.538116 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-workload-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.538426 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.538173 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-workload-certs\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.538426 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.538263 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5ldd\" (UniqueName: \"kubernetes.io/projected/d6978eea-f24f-4d30-83b0-85b0015baf53-kube-api-access-d5ldd\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.538596 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.538477 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-data\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.538760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.538732 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-workload-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.538926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.538903 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-workload-certs\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.539049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.538851 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6978eea-f24f-4d30-83b0-85b0015baf53-istiod-ca-cert\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.539049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.538806 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-credential-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.540794 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.540773 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-envoy\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.540919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.540902 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-podinfo\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.598863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.598841 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-token\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.599210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.599189 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5ldd\" (UniqueName: \"kubernetes.io/projected/d6978eea-f24f-4d30-83b0-85b0015baf53-kube-api-access-d5ldd\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-c9x78\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:41.619182 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:41.619159 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:42.055264 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:05:42.055234 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6978eea_f24f_4d30_83b0_85b0015baf53.slice/crio-4e14eb3d811f5d9f97738c0dcebfa5983dab317570d00ce5e383c3de9db89fa1 WatchSource:0}: Error finding container 4e14eb3d811f5d9f97738c0dcebfa5983dab317570d00ce5e383c3de9db89fa1: Status 404 returned error can't find the container with id 4e14eb3d811f5d9f97738c0dcebfa5983dab317570d00ce5e383c3de9db89fa1 Oct 30 22:05:42.055933 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:42.055913 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78"] Oct 30 22:05:42.057509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:42.057480 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:05:42.057610 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:42.057539 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:05:42.057610 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:42.057578 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:05:42.083831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:42.083808 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" event={"ID":"d6978eea-f24f-4d30-83b0-85b0015baf53","Type":"ContainerStarted","Data":"4e14eb3d811f5d9f97738c0dcebfa5983dab317570d00ce5e383c3de9db89fa1"} Oct 30 22:05:43.092253 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:43.092218 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" event={"ID":"d6978eea-f24f-4d30-83b0-85b0015baf53","Type":"ContainerStarted","Data":"3d4598e61d58fabf8822a679c60dcfa7e9e7dd21f36f77a7acfaee709fc73041"} Oct 30 22:05:43.329150 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:43.329077 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" podStartSLOduration=2.32905465 podStartE2EDuration="2.32905465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:43.260005598 +0000 UTC m=+1276.360437330" watchObservedRunningTime="2026-10-30 22:05:43.32905465 +0000 UTC m=+1276.429486382" Oct 30 22:05:43.621643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:43.621355 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:43.626679 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:43.626564 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-c9x78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:05:43.626679 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:43.626650 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" podUID="d6978eea-f24f-4d30-83b0-85b0015baf53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 22:05:44.626641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:44.622919 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-c9x78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:05:44.626641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:44.622996 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" podUID="d6978eea-f24f-4d30-83b0-85b0015baf53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 22:05:45.622995 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:45.622956 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-c9x78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:05:45.623306 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:45.623277 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" podUID="d6978eea-f24f-4d30-83b0-85b0015baf53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 22:05:46.627238 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:46.627206 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-c9x78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:05:46.627789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:46.627325 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" podUID="d6978eea-f24f-4d30-83b0-85b0015baf53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 22:05:47.620014 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:47.619976 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-c9x78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:05:47.620296 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:47.620266 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" podUID="d6978eea-f24f-4d30-83b0-85b0015baf53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 22:05:48.619857 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:48.619820 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-c9x78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:05:48.620379 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:48.619868 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" podUID="d6978eea-f24f-4d30-83b0-85b0015baf53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 22:05:49.623871 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:49.623842 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:49.624304 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:49.624288 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:49.624876 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:05:49.624860 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:05:55.321985 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:05:55.321945 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:06:15.709603 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.709527 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd"] Oct 30 22:06:15.713678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.713662 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:15.816236 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.816207 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a7b06b\"/\"openshift-service-ca.crt\"" Oct 30 22:06:15.816421 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.816297 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a7b06b\"/\"kube-root-ca.crt\"" Oct 30 22:06:15.816421 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.816413 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a7b06b\"/\"distinct-w5-a7b06b-kserve-self-signed-certs\"" Oct 30 22:06:15.816503 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.816434 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a7b06b\"/\"default-dockercfg-82vdb\"" Oct 30 22:06:15.818952 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.818930 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd"] Oct 30 22:06:15.883029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.882959 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-tls-certs\") pod \"distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:15.883381 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.883358 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p7sl\" (UniqueName: \"kubernetes.io/projected/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-kube-api-access-4p7sl\") pod \"distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:15.883517 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.883495 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-model-cache\") pod \"distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:15.883873 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.883845 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-tmp-dir\") pod \"distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:15.884114 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.884098 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-dshm\") pod \"distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:15.884255 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.884241 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-home\") pod \"distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:15.985773 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.985685 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-tmp-dir\") pod \"distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:15.985773 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.985744 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-dshm\") pod \"distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:15.985990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.985780 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-home\") pod \"distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:15.985990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.985817 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-tls-certs\") pod \"distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:15.985990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.985891 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p7sl\" (UniqueName: \"kubernetes.io/projected/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-kube-api-access-4p7sl\") pod \"distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:15.985990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.985925 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-model-cache\") pod \"distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:15.986216 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.986182 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-tmp-dir\") pod \"distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:15.986292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.986223 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-home\") pod \"distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:15.986372 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.986348 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-model-cache\") pod \"distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:15.988391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.988371 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-dshm\") pod \"distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:15.988628 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:15.988593 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-tls-certs\") pod \"distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:16.048430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:16.048405 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p7sl\" (UniqueName: \"kubernetes.io/projected/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-kube-api-access-4p7sl\") pod \"distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:16.327223 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:16.327191 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:16.711137 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:06:16.711100 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4eefc8ca_80bf_4757_91ff_ef5d8ebffa23.slice/crio-1d55a0e6d60d1dc21a7372b6e93fd86709c14b37f012ef1c80397a00226b526b WatchSource:0}: Error finding container 1d55a0e6d60d1dc21a7372b6e93fd86709c14b37f012ef1c80397a00226b526b: Status 404 returned error can't find the container with id 1d55a0e6d60d1dc21a7372b6e93fd86709c14b37f012ef1c80397a00226b526b Oct 30 22:06:16.711665 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:16.711644 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd"] Oct 30 22:06:17.395791 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:17.395757 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" event={"ID":"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23","Type":"ContainerStarted","Data":"23faa20178757dd3234fef0531352b0869bc850769d0a3cae07e081ef4ce0157"} Oct 30 22:06:17.395970 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:17.395805 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" event={"ID":"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23","Type":"ContainerStarted","Data":"1d55a0e6d60d1dc21a7372b6e93fd86709c14b37f012ef1c80397a00226b526b"} Oct 30 22:06:17.515471 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:17.515428 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podStartSLOduration=2.515414978 podStartE2EDuration="2.515414978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:17.511235252 +0000 UTC m=+1310.611666985" watchObservedRunningTime="2026-10-30 22:06:17.515414978 +0000 UTC m=+1310.615846711" Oct 30 22:06:18.327795 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:18.327765 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:25.324515 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:06:25.324473 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:06:26.331099 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:26.331062 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:26.360397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:26.360371 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:26.497948 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:26.497920 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:06:28.953537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:28.953498 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:06:30.469911 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.469870 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6"] Oct 30 22:06:30.477473 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.477442 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.586635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.586260 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-podinfo\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.586635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.586357 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-workload-certs\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.586635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.586401 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-envoy\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.586635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.586435 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c520ab-1d01-497c-9ac9-a10bfd284e54-istiod-ca-cert\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.586635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.586467 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-workload-socket\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.586635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.586510 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5rfq\" (UniqueName: \"kubernetes.io/projected/63c520ab-1d01-497c-9ac9-a10bfd284e54-kube-api-access-j5rfq\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.586635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.586549 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-token\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.586635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.586631 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-credential-socket\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.587236 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.586658 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-data\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.588681 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.587679 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6"] Oct 30 22:06:30.688207 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.688168 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-token\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.688396 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.688246 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-credential-socket\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.688396 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.688275 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-data\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.688396 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.688351 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-podinfo\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.688579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.688434 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-workload-certs\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.688579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.688482 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-envoy\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.688579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.688520 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c520ab-1d01-497c-9ac9-a10bfd284e54-istiod-ca-cert\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.688579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.688561 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-workload-socket\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.688888 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.688602 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5rfq\" (UniqueName: \"kubernetes.io/projected/63c520ab-1d01-497c-9ac9-a10bfd284e54-kube-api-access-j5rfq\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.688888 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.688727 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-credential-socket\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.688888 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.688758 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-data\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.688888 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.688791 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-workload-certs\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.689021 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.688956 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-workload-socket\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.689416 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.689390 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c520ab-1d01-497c-9ac9-a10bfd284e54-istiod-ca-cert\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.691110 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.691082 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-podinfo\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.691582 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.691556 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-envoy\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.763548 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.763489 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-token\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.763835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.763809 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5rfq\" (UniqueName: \"kubernetes.io/projected/63c520ab-1d01-497c-9ac9-a10bfd284e54-kube-api-access-j5rfq\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:30.795805 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:30.795780 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:31.251949 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:06:31.251914 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c520ab_1d01_497c_9ac9_a10bfd284e54.slice/crio-9d282f1aa4f21345d8c6f459fe924046cbb684bc6cecd00f2aad7631d8783c33 WatchSource:0}: Error finding container 9d282f1aa4f21345d8c6f459fe924046cbb684bc6cecd00f2aad7631d8783c33: Status 404 returned error can't find the container with id 9d282f1aa4f21345d8c6f459fe924046cbb684bc6cecd00f2aad7631d8783c33 Oct 30 22:06:31.254187 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:31.254011 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6"] Oct 30 22:06:31.255043 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:31.255009 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:06:31.255165 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:31.255085 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:06:31.255165 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:31.255130 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:06:31.619204 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:31.619175 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" event={"ID":"63c520ab-1d01-497c-9ac9-a10bfd284e54","Type":"ContainerStarted","Data":"b9de59cda9a88634dc048d8d522d60cb62d886f7df20ae5d365cbd3ce13aa007"} Oct 30 22:06:31.619722 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:31.619232 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" event={"ID":"63c520ab-1d01-497c-9ac9-a10bfd284e54","Type":"ContainerStarted","Data":"9d282f1aa4f21345d8c6f459fe924046cbb684bc6cecd00f2aad7631d8783c33"} Oct 30 22:06:31.746050 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:31.745847 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" podStartSLOduration=1.7458265499999999 podStartE2EDuration="1.74582655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:31.742116427 +0000 UTC m=+1324.842548160" watchObservedRunningTime="2026-10-30 22:06:31.74582655 +0000 UTC m=+1324.846258283" Oct 30 22:06:31.796138 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:31.796047 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:32.819639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:32.818721 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:06:32.819639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:32.818783 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" podUID="63c520ab-1d01-497c-9ac9-a10bfd284e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 22:06:33.796945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:33.796910 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:06:33.797199 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:33.797170 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" podUID="63c520ab-1d01-497c-9ac9-a10bfd284e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 22:06:34.800652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:34.799904 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:06:34.800652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:34.799975 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" podUID="63c520ab-1d01-497c-9ac9-a10bfd284e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 22:06:35.799947 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:35.799911 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:06:35.800873 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:35.800842 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" podUID="63c520ab-1d01-497c-9ac9-a10bfd284e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 22:06:36.811558 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:36.811526 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:06:36.812165 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:36.812133 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" podUID="63c520ab-1d01-497c-9ac9-a10bfd284e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 22:06:37.796211 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:37.796175 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:06:37.796421 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:37.796244 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" podUID="63c520ab-1d01-497c-9ac9-a10bfd284e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 22:06:38.800811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:38.800780 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:38.801294 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:38.801277 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:38.802021 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:38.801997 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:06:55.322675 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:06:55.322638 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:06:58.773423 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:58.773388 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd"] Oct 30 22:06:58.777942 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:58.777908 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd" Oct 30 22:06:58.830589 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:58.830564 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:06:58.836180 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:58.836157 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd"] Oct 30 22:06:58.848847 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:58.848818 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba146f29-2902-499d-9c92-93b1cb3beeb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b84145f-4xqvd\" (UID: \"ba146f29-2902-499d-9c92-93b1cb3beeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd" Oct 30 22:06:58.849024 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:58.848998 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba146f29-2902-499d-9c92-93b1cb3beeb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b84145f-4xqvd\" (UID: \"ba146f29-2902-499d-9c92-93b1cb3beeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd" Oct 30 22:06:58.849203 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:58.849177 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c4ft\" (UniqueName: \"kubernetes.io/projected/ba146f29-2902-499d-9c92-93b1cb3beeb6-kube-api-access-2c4ft\") pod \"maas-api-revoke-sub-3b84145f-4xqvd\" (UID: \"ba146f29-2902-499d-9c92-93b1cb3beeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd" Oct 30 22:06:58.878783 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:58.878743 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:06:58.878887 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:58.878806 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:06:58.880018 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:58.879996 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:06:58.950465 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:58.950442 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba146f29-2902-499d-9c92-93b1cb3beeb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b84145f-4xqvd\" (UID: \"ba146f29-2902-499d-9c92-93b1cb3beeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd" Oct 30 22:06:58.950579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:58.950521 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba146f29-2902-499d-9c92-93b1cb3beeb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b84145f-4xqvd\" (UID: \"ba146f29-2902-499d-9c92-93b1cb3beeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd" Oct 30 22:06:58.950667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:58.950650 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c4ft\" (UniqueName: \"kubernetes.io/projected/ba146f29-2902-499d-9c92-93b1cb3beeb6-kube-api-access-2c4ft\") pod \"maas-api-revoke-sub-3b84145f-4xqvd\" (UID: \"ba146f29-2902-499d-9c92-93b1cb3beeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd" Oct 30 22:06:58.951218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:58.951199 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba146f29-2902-499d-9c92-93b1cb3beeb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b84145f-4xqvd\" (UID: \"ba146f29-2902-499d-9c92-93b1cb3beeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd" Oct 30 22:06:58.953004 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:58.952981 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba146f29-2902-499d-9c92-93b1cb3beeb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b84145f-4xqvd\" (UID: \"ba146f29-2902-499d-9c92-93b1cb3beeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd" Oct 30 22:06:59.012394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:59.012367 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c4ft\" (UniqueName: \"kubernetes.io/projected/ba146f29-2902-499d-9c92-93b1cb3beeb6-kube-api-access-2c4ft\") pod \"maas-api-revoke-sub-3b84145f-4xqvd\" (UID: \"ba146f29-2902-499d-9c92-93b1cb3beeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd" Oct 30 22:06:59.092798 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:59.092773 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd" Oct 30 22:06:59.532545 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:06:59.532503 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba146f29_2902_499d_9c92_93b1cb3beeb6.slice/crio-715fc3bc4b583de997372fdc0246f4b8329e1f000d748e4f4fa32b88c5872246 WatchSource:0}: Error finding container 715fc3bc4b583de997372fdc0246f4b8329e1f000d748e4f4fa32b88c5872246: Status 404 returned error can't find the container with id 715fc3bc4b583de997372fdc0246f4b8329e1f000d748e4f4fa32b88c5872246 Oct 30 22:06:59.534352 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:59.534326 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd"] Oct 30 22:06:59.583229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:59.583194 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:06:59.881125 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:59.881088 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd" event={"ID":"ba146f29-2902-499d-9c92-93b1cb3beeb6","Type":"ContainerStarted","Data":"2775e31261cb74bef7856a04a29d3d5d2497b0349908a4e79aea3d3f2827d607"} Oct 30 22:06:59.881517 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:06:59.881148 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd" event={"ID":"ba146f29-2902-499d-9c92-93b1cb3beeb6","Type":"ContainerStarted","Data":"715fc3bc4b583de997372fdc0246f4b8329e1f000d748e4f4fa32b88c5872246"} Oct 30 22:07:00.001270 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:00.001221 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd" podStartSLOduration=2.001207604 podStartE2EDuration="2.001207604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:59.995997149 +0000 UTC m=+1353.096428895" watchObservedRunningTime="2026-10-30 22:07:00.001207604 +0000 UTC m=+1353.101639335" Oct 30 22:07:00.892889 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:00.892852 4335 generic.go:356] "Generic (PLEG): container finished" podID="ba146f29-2902-499d-9c92-93b1cb3beeb6" containerID="2775e31261cb74bef7856a04a29d3d5d2497b0349908a4e79aea3d3f2827d607" exitCode=0 Oct 30 22:07:00.893466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:00.892937 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd" event={"ID":"ba146f29-2902-499d-9c92-93b1cb3beeb6","Type":"ContainerDied","Data":"2775e31261cb74bef7856a04a29d3d5d2497b0349908a4e79aea3d3f2827d607"} Oct 30 22:07:01.078185 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.078160 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq"] Oct 30 22:07:01.082984 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.082961 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.184474 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.184396 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-token\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.184474 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.184459 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z84mn\" (UniqueName: \"kubernetes.io/projected/87f6d9bc-feca-46a3-925f-e3402a1d895c-kube-api-access-z84mn\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.184678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.184525 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-data\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.184678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.184556 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-envoy\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.184774 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.184675 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-credential-socket\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.184774 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.184750 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-workload-socket\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.184853 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.184831 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87f6d9bc-feca-46a3-925f-e3402a1d895c-istiod-ca-cert\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.184891 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.184867 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-workload-certs\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.184933 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.184891 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-podinfo\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.185627 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.185587 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq"] Oct 30 22:07:01.286712 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.286673 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-token\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.286870 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.286737 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z84mn\" (UniqueName: \"kubernetes.io/projected/87f6d9bc-feca-46a3-925f-e3402a1d895c-kube-api-access-z84mn\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.286870 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.286804 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-data\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.286870 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.286834 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-envoy\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.286994 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.286919 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-credential-socket\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.287043 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.286998 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-workload-socket\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.287101 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.287085 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87f6d9bc-feca-46a3-925f-e3402a1d895c-istiod-ca-cert\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.287158 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.287126 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-workload-certs\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.287158 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.287151 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-podinfo\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.287258 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.287236 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-data\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.287603 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.287574 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-workload-socket\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.287774 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.287644 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-credential-socket\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.287898 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.287871 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-workload-certs\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.287961 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.287887 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87f6d9bc-feca-46a3-925f-e3402a1d895c-istiod-ca-cert\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.289358 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.289336 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-envoy\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.289829 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.289808 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-podinfo\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.352356 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.352328 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z84mn\" (UniqueName: \"kubernetes.io/projected/87f6d9bc-feca-46a3-925f-e3402a1d895c-kube-api-access-z84mn\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.361541 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.361516 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-token\") pod \"e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.397509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.397484 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:01.872406 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:07:01.872376 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f6d9bc_feca_46a3_925f_e3402a1d895c.slice/crio-63dc384d09d9db0892ba0d514aa74319f66d7d65aea81bb5e4ee82199a808d65 WatchSource:0}: Error finding container 63dc384d09d9db0892ba0d514aa74319f66d7d65aea81bb5e4ee82199a808d65: Status 404 returned error can't find the container with id 63dc384d09d9db0892ba0d514aa74319f66d7d65aea81bb5e4ee82199a808d65 Oct 30 22:07:01.873885 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.873860 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq"] Oct 30 22:07:01.875386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.875355 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:07:01.875480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.875432 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:07:01.875548 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.875477 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:07:01.904176 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:01.904144 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" event={"ID":"87f6d9bc-feca-46a3-925f-e3402a1d895c","Type":"ContainerStarted","Data":"63dc384d09d9db0892ba0d514aa74319f66d7d65aea81bb5e4ee82199a808d65"} Oct 30 22:07:02.086400 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:02.086374 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd" Oct 30 22:07:02.207408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:02.202184 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba146f29-2902-499d-9c92-93b1cb3beeb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba146f29-2902-499d-9c92-93b1cb3beeb6-maas-api-xks-ca\") pod \"ba146f29-2902-499d-9c92-93b1cb3beeb6\" (UID: \"ba146f29-2902-499d-9c92-93b1cb3beeb6\") " Oct 30 22:07:02.207408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:02.202434 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba146f29-2902-499d-9c92-93b1cb3beeb6-kube-api-access-2c4ft\" (UniqueName: \"kubernetes.io/projected/ba146f29-2902-499d-9c92-93b1cb3beeb6-kube-api-access-2c4ft\") pod \"ba146f29-2902-499d-9c92-93b1cb3beeb6\" (UID: \"ba146f29-2902-499d-9c92-93b1cb3beeb6\") " Oct 30 22:07:02.207408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:02.202486 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba146f29-2902-499d-9c92-93b1cb3beeb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba146f29-2902-499d-9c92-93b1cb3beeb6-maas-api-openshift-ca\") pod \"ba146f29-2902-499d-9c92-93b1cb3beeb6\" (UID: \"ba146f29-2902-499d-9c92-93b1cb3beeb6\") " Oct 30 22:07:02.207408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:02.203170 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba146f29-2902-499d-9c92-93b1cb3beeb6-maas-api-openshift-ca" pod "ba146f29-2902-499d-9c92-93b1cb3beeb6" (UID: "ba146f29-2902-499d-9c92-93b1cb3beeb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:02.207408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:02.203509 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba146f29-2902-499d-9c92-93b1cb3beeb6-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:02.207408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:02.205690 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba146f29-2902-499d-9c92-93b1cb3beeb6-kube-api-access-2c4ft" pod "ba146f29-2902-499d-9c92-93b1cb3beeb6" (UID: "ba146f29-2902-499d-9c92-93b1cb3beeb6"). InnerVolumeSpecName "kube-api-access-2c4ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:02.207408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:02.206728 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba146f29-2902-499d-9c92-93b1cb3beeb6-maas-api-xks-ca" pod "ba146f29-2902-499d-9c92-93b1cb3beeb6" (UID: "ba146f29-2902-499d-9c92-93b1cb3beeb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:02.305029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:02.304999 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c4ft\" (UniqueName: \"kubernetes.io/projected/ba146f29-2902-499d-9c92-93b1cb3beeb6-kube-api-access-2c4ft\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:02.305151 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:02.305032 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba146f29-2902-499d-9c92-93b1cb3beeb6-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:02.936034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:02.936001 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd" Oct 30 22:07:02.936034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:02.936019 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd" event={"ID":"ba146f29-2902-499d-9c92-93b1cb3beeb6","Type":"ContainerDied","Data":"715fc3bc4b583de997372fdc0246f4b8329e1f000d748e4f4fa32b88c5872246"} Oct 30 22:07:02.936586 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:02.936097 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="715fc3bc4b583de997372fdc0246f4b8329e1f000d748e4f4fa32b88c5872246" Oct 30 22:07:02.938047 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:02.937988 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" event={"ID":"87f6d9bc-feca-46a3-925f-e3402a1d895c","Type":"ContainerStarted","Data":"8ab4780120d00a95318452be76e9c280b9fcda0051fb790f7a33adf475ceed1e"} Oct 30 22:07:03.063744 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:03.063686 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" podStartSLOduration=2.06366752 podStartE2EDuration="2.06366752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:03.057959503 +0000 UTC m=+1356.158391256" watchObservedRunningTime="2026-10-30 22:07:03.06366752 +0000 UTC m=+1356.164099253" Oct 30 22:07:03.413640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:03.403974 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:03.413640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:03.406479 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:07:03.413640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:03.406534 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" podUID="87f6d9bc-feca-46a3-925f-e3402a1d895c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 22:07:04.404642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:04.403471 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:07:04.404642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:04.403533 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" podUID="87f6d9bc-feca-46a3-925f-e3402a1d895c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 22:07:05.411640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:05.399950 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:07:05.411640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:05.400009 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" podUID="87f6d9bc-feca-46a3-925f-e3402a1d895c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 22:07:06.415934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:06.415897 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:07:06.416492 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:06.416464 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" podUID="87f6d9bc-feca-46a3-925f-e3402a1d895c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 22:07:07.412643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:07.399978 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:07:07.412643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:07.400054 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" podUID="87f6d9bc-feca-46a3-925f-e3402a1d895c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 22:07:08.398828 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:08.398793 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:07:08.399211 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:08.398848 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" podUID="87f6d9bc-feca-46a3-925f-e3402a1d895c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 22:07:09.402571 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:09.402540 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:09.403098 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:09.403077 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:09.403864 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:09.403843 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:07:18.067786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:18.067751 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b"] Oct 30 22:07:18.115958 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:18.115930 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af36e54a-vbm2b"] Oct 30 22:07:19.058313 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:19.058282 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee15a032-782e-4441-a3df-8352743ba9fc" path="/var/lib/kubelet/pods/ee15a032-782e-4441-a3df-8352743ba9fc/volumes" Oct 30 22:07:19.995355 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:19.995320 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn"] Oct 30 22:07:19.996124 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:19.996103 4335 cpu_manager.go:414] "containerMap: removing container" podUID="ba146f29-2902-499d-9c92-93b1cb3beeb6" containerName="revoke-keys" Oct 30 22:07:19.996221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:19.996125 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba146f29-2902-499d-9c92-93b1cb3beeb6" containerName="revoke-keys" Oct 30 22:07:19.996317 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:19.996303 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba146f29-2902-499d-9c92-93b1cb3beeb6" containerName="revoke-keys" Oct 30 22:07:19.999906 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:19.999885 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.052055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.052029 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-bf7cbf\"/\"distinct-2-w1-bf7cbf-kserve-self-signed-certs\"" Oct 30 22:07:20.058540 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.058512 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn"] Oct 30 22:07:20.117048 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.117025 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6173209e-9700-444d-b33a-cabb2d51816a-tls-certs\") pod \"distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.117217 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.117197 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-dshm\") pod \"distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.117270 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.117244 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-home\") pod \"distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.117270 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.117261 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-model-cache\") pod \"distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.117379 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.117362 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6qcz\" (UniqueName: \"kubernetes.io/projected/6173209e-9700-444d-b33a-cabb2d51816a-kube-api-access-b6qcz\") pod \"distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.117412 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.117391 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-tmp-dir\") pod \"distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.219047 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.219015 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-dshm\") pod \"distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.219207 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.219076 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-home\") pod \"distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.219207 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.219092 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-model-cache\") pod \"distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.219207 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.219136 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6qcz\" (UniqueName: \"kubernetes.io/projected/6173209e-9700-444d-b33a-cabb2d51816a-kube-api-access-b6qcz\") pod \"distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.219207 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.219166 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-tmp-dir\") pod \"distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.219476 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.219390 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6173209e-9700-444d-b33a-cabb2d51816a-tls-certs\") pod \"distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.219536 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.219502 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-home\") pod \"distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.219606 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.219587 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-tmp-dir\") pod \"distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.219808 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.219790 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-model-cache\") pod \"distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.221563 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.221536 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-dshm\") pod \"distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.222005 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.221987 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6173209e-9700-444d-b33a-cabb2d51816a-tls-certs\") pod \"distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.280283 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.280232 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6qcz\" (UniqueName: \"kubernetes.io/projected/6173209e-9700-444d-b33a-cabb2d51816a-kube-api-access-b6qcz\") pod \"distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.312953 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.312931 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:20.730779 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:20.730748 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn"] Oct 30 22:07:20.731024 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:07:20.731001 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6173209e_9700_444d_b33a_cabb2d51816a.slice/crio-f9860d77b43777d8e96ac669b1ff880153b2ede7a5f718c43f43986cfd3f6da0 WatchSource:0}: Error finding container f9860d77b43777d8e96ac669b1ff880153b2ede7a5f718c43f43986cfd3f6da0: Status 404 returned error can't find the container with id f9860d77b43777d8e96ac669b1ff880153b2ede7a5f718c43f43986cfd3f6da0 Oct 30 22:07:21.094729 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:21.094694 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" event={"ID":"6173209e-9700-444d-b33a-cabb2d51816a","Type":"ContainerStarted","Data":"1701349ffec22d1b0ac48a1e52313d7fdfc0cb4d3bb6956f8ab2d299a4486f03"} Oct 30 22:07:21.095080 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:21.094738 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" event={"ID":"6173209e-9700-444d-b33a-cabb2d51816a","Type":"ContainerStarted","Data":"f9860d77b43777d8e96ac669b1ff880153b2ede7a5f718c43f43986cfd3f6da0"} Oct 30 22:07:21.213018 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:21.212966 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podStartSLOduration=2.212951988 podStartE2EDuration="2.212951988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:21.210679934 +0000 UTC m=+1374.311111665" watchObservedRunningTime="2026-10-30 22:07:21.212951988 +0000 UTC m=+1374.313383720" Oct 30 22:07:21.313034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:21.313004 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:25.322423 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:07:25.322382 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:07:30.313676 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:30.313640 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:30.325884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:30.325855 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:31.211811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:31.211783 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:07:37.912910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:37.912880 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8"] Oct 30 22:07:37.918271 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:37.918254 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:37.977118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:37.977089 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8"] Oct 30 22:07:38.013423 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.013393 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-tmp-dir\") pod \"distinct-2-w4-536975-kserve-85946c9cf5-djhz8\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.013590 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.013457 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-model-cache\") pod \"distinct-2-w4-536975-kserve-85946c9cf5-djhz8\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.013683 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.013641 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-home\") pod \"distinct-2-w4-536975-kserve-85946c9cf5-djhz8\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.013745 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.013686 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4a9040-3e1b-4c8c-b532-36140681da06-tls-certs\") pod \"distinct-2-w4-536975-kserve-85946c9cf5-djhz8\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.013800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.013757 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-dshm\") pod \"distinct-2-w4-536975-kserve-85946c9cf5-djhz8\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.013800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.013792 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwfwb\" (UniqueName: \"kubernetes.io/projected/7c4a9040-3e1b-4c8c-b532-36140681da06-kube-api-access-wwfwb\") pod \"distinct-2-w4-536975-kserve-85946c9cf5-djhz8\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.019298 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.019274 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-536975\"/\"distinct-2-w4-536975-kserve-self-signed-certs\"" Oct 30 22:07:38.116674 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.114911 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-tmp-dir\") pod \"distinct-2-w4-536975-kserve-85946c9cf5-djhz8\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.116674 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.114996 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-model-cache\") pod \"distinct-2-w4-536975-kserve-85946c9cf5-djhz8\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.116674 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.115116 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-home\") pod \"distinct-2-w4-536975-kserve-85946c9cf5-djhz8\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.116674 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.115146 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4a9040-3e1b-4c8c-b532-36140681da06-tls-certs\") pod \"distinct-2-w4-536975-kserve-85946c9cf5-djhz8\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.116674 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.115267 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-dshm\") pod \"distinct-2-w4-536975-kserve-85946c9cf5-djhz8\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.116674 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.115297 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwfwb\" (UniqueName: \"kubernetes.io/projected/7c4a9040-3e1b-4c8c-b532-36140681da06-kube-api-access-wwfwb\") pod \"distinct-2-w4-536975-kserve-85946c9cf5-djhz8\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.116674 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.116113 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-home\") pod \"distinct-2-w4-536975-kserve-85946c9cf5-djhz8\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.116674 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.116422 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-tmp-dir\") pod \"distinct-2-w4-536975-kserve-85946c9cf5-djhz8\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.117263 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.116698 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-model-cache\") pod \"distinct-2-w4-536975-kserve-85946c9cf5-djhz8\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.119415 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.119391 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-dshm\") pod \"distinct-2-w4-536975-kserve-85946c9cf5-djhz8\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.124165 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.124134 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4a9040-3e1b-4c8c-b532-36140681da06-tls-certs\") pod \"distinct-2-w4-536975-kserve-85946c9cf5-djhz8\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.177268 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.177179 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwfwb\" (UniqueName: \"kubernetes.io/projected/7c4a9040-3e1b-4c8c-b532-36140681da06-kube-api-access-wwfwb\") pod \"distinct-2-w4-536975-kserve-85946c9cf5-djhz8\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.231794 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.231768 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:38.637002 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:07:38.636968 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c4a9040_3e1b_4c8c_b532_36140681da06.slice/crio-8a87cf84b734669937a2c0871905e441ee9dc06b27dbf4afc56b5c3527276f6a WatchSource:0}: Error finding container 8a87cf84b734669937a2c0871905e441ee9dc06b27dbf4afc56b5c3527276f6a: Status 404 returned error can't find the container with id 8a87cf84b734669937a2c0871905e441ee9dc06b27dbf4afc56b5c3527276f6a Oct 30 22:07:38.638061 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:38.638034 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8"] Oct 30 22:07:39.270486 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:39.270454 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" event={"ID":"7c4a9040-3e1b-4c8c-b532-36140681da06","Type":"ContainerStarted","Data":"5df9f26475e5c5bc2e65c78008d1bdccfa3218ae11c8071770b1cb0b522ef702"} Oct 30 22:07:39.270486 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:39.270492 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" event={"ID":"7c4a9040-3e1b-4c8c-b532-36140681da06","Type":"ContainerStarted","Data":"8a87cf84b734669937a2c0871905e441ee9dc06b27dbf4afc56b5c3527276f6a"} Oct 30 22:07:39.387257 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:39.387207 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podStartSLOduration=2.387194493 podStartE2EDuration="2.387194493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:39.386574107 +0000 UTC m=+1392.487005843" watchObservedRunningTime="2026-10-30 22:07:39.387194493 +0000 UTC m=+1392.487626225" Oct 30 22:07:40.232030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:40.231996 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:42.214509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:42.214477 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:07:44.053863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:44.053818 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4"] Oct 30 22:07:44.099010 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:44.098982 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1acdff-prtl4"] Oct 30 22:07:45.058548 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:45.058511 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a026eb-0f67-4844-bf37-ca7b42807e0b" path="/var/lib/kubelet/pods/00a026eb-0f67-4844-bf37-ca7b42807e0b/volumes" Oct 30 22:07:48.231977 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:48.231942 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:48.245072 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:48.245051 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:48.366132 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:07:48.366107 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:07:55.321477 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:07:55.321440 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:08:00.051823 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:00.051790 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf"] Oct 30 22:08:00.101312 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:00.101284 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b06dfc7d-lqgsf"] Oct 30 22:08:00.608774 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:00.608745 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc"] Oct 30 22:08:00.613343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:00.613321 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc" Oct 30 22:08:00.670382 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:00.670351 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc"] Oct 30 22:08:00.714221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:00.714196 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:08:00.714376 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:00.714200 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:00.714376 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:00.714203 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:08:00.714376 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:00.714260 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:08:00.789980 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:00.789952 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc8371e7-7efd-4728-8a3b-82ce85f361c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-019b41c8-cxwjc\" (UID: \"bc8371e7-7efd-4728-8a3b-82ce85f361c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc" Oct 30 22:08:00.790139 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:00.790009 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc8371e7-7efd-4728-8a3b-82ce85f361c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-019b41c8-cxwjc\" (UID: \"bc8371e7-7efd-4728-8a3b-82ce85f361c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc" Oct 30 22:08:00.790182 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:00.790159 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9j8n\" (UniqueName: \"kubernetes.io/projected/bc8371e7-7efd-4728-8a3b-82ce85f361c9-kube-api-access-d9j8n\") pod \"maas-api-revoke-sub-019b41c8-cxwjc\" (UID: \"bc8371e7-7efd-4728-8a3b-82ce85f361c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc" Oct 30 22:08:00.891390 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:00.891306 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9j8n\" (UniqueName: \"kubernetes.io/projected/bc8371e7-7efd-4728-8a3b-82ce85f361c9-kube-api-access-d9j8n\") pod \"maas-api-revoke-sub-019b41c8-cxwjc\" (UID: \"bc8371e7-7efd-4728-8a3b-82ce85f361c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc" Oct 30 22:08:00.891545 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:00.891425 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc8371e7-7efd-4728-8a3b-82ce85f361c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-019b41c8-cxwjc\" (UID: \"bc8371e7-7efd-4728-8a3b-82ce85f361c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc" Oct 30 22:08:00.891545 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:00.891473 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc8371e7-7efd-4728-8a3b-82ce85f361c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-019b41c8-cxwjc\" (UID: \"bc8371e7-7efd-4728-8a3b-82ce85f361c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc" Oct 30 22:08:00.892193 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:00.892170 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc8371e7-7efd-4728-8a3b-82ce85f361c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-019b41c8-cxwjc\" (UID: \"bc8371e7-7efd-4728-8a3b-82ce85f361c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc" Oct 30 22:08:00.893998 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:00.893973 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc8371e7-7efd-4728-8a3b-82ce85f361c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-019b41c8-cxwjc\" (UID: \"bc8371e7-7efd-4728-8a3b-82ce85f361c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc" Oct 30 22:08:00.951539 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:00.951509 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9j8n\" (UniqueName: \"kubernetes.io/projected/bc8371e7-7efd-4728-8a3b-82ce85f361c9-kube-api-access-d9j8n\") pod \"maas-api-revoke-sub-019b41c8-cxwjc\" (UID: \"bc8371e7-7efd-4728-8a3b-82ce85f361c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc" Oct 30 22:08:01.061523 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:01.061490 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08eb5248-5ccb-4202-baed-c2c0651f3cfb" path="/var/lib/kubelet/pods/08eb5248-5ccb-4202-baed-c2c0651f3cfb/volumes" Oct 30 22:08:01.061995 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:01.061973 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq"] Oct 30 22:08:01.063536 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:01.063514 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd935447-4knkq"] Oct 30 22:08:01.229364 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:01.229289 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc" Oct 30 22:08:01.615277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:01.615249 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc"] Oct 30 22:08:01.615450 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:08:01.615412 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8371e7_7efd_4728_8a3b_82ce85f361c9.slice/crio-32d782290359e83f7ebe1e4844d39c8109e55e14951d53e42ed95cdfdf1a21e4 WatchSource:0}: Error finding container 32d782290359e83f7ebe1e4844d39c8109e55e14951d53e42ed95cdfdf1a21e4: Status 404 returned error can't find the container with id 32d782290359e83f7ebe1e4844d39c8109e55e14951d53e42ed95cdfdf1a21e4 Oct 30 22:08:02.533639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:02.522255 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc" event={"ID":"bc8371e7-7efd-4728-8a3b-82ce85f361c9","Type":"ContainerStarted","Data":"50a0a6363276f011b13bec0ce6d5fdfdae89c078dafc77bf57b86f94528bd9da"} Oct 30 22:08:02.533639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:02.522331 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc" event={"ID":"bc8371e7-7efd-4728-8a3b-82ce85f361c9","Type":"ContainerStarted","Data":"32d782290359e83f7ebe1e4844d39c8109e55e14951d53e42ed95cdfdf1a21e4"} Oct 30 22:08:02.641675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:02.641576 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc" podStartSLOduration=2.641556605 podStartE2EDuration="2.641556605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:02.640665318 +0000 UTC m=+1415.741097053" watchObservedRunningTime="2026-10-30 22:08:02.641556605 +0000 UTC m=+1415.741988339" Oct 30 22:08:03.122876 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:03.122838 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93b43447-cccf-4ecc-9535-26e54a96c27b" path="/var/lib/kubelet/pods/93b43447-cccf-4ecc-9535-26e54a96c27b/volumes" Oct 30 22:08:03.605640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:03.603795 4335 generic.go:356] "Generic (PLEG): container finished" podID="bc8371e7-7efd-4728-8a3b-82ce85f361c9" containerID="50a0a6363276f011b13bec0ce6d5fdfdae89c078dafc77bf57b86f94528bd9da" exitCode=0 Oct 30 22:08:03.605640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:03.603949 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc" event={"ID":"bc8371e7-7efd-4728-8a3b-82ce85f361c9","Type":"ContainerDied","Data":"50a0a6363276f011b13bec0ce6d5fdfdae89c078dafc77bf57b86f94528bd9da"} Oct 30 22:08:04.774835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:04.774813 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc" Oct 30 22:08:04.888099 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:04.888014 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc8371e7-7efd-4728-8a3b-82ce85f361c9-kube-api-access-d9j8n\" (UniqueName: \"kubernetes.io/projected/bc8371e7-7efd-4728-8a3b-82ce85f361c9-kube-api-access-d9j8n\") pod \"bc8371e7-7efd-4728-8a3b-82ce85f361c9\" (UID: \"bc8371e7-7efd-4728-8a3b-82ce85f361c9\") " Oct 30 22:08:04.888265 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:04.888132 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc8371e7-7efd-4728-8a3b-82ce85f361c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc8371e7-7efd-4728-8a3b-82ce85f361c9-maas-api-openshift-ca\") pod \"bc8371e7-7efd-4728-8a3b-82ce85f361c9\" (UID: \"bc8371e7-7efd-4728-8a3b-82ce85f361c9\") " Oct 30 22:08:04.888325 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:04.888260 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc8371e7-7efd-4728-8a3b-82ce85f361c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc8371e7-7efd-4728-8a3b-82ce85f361c9-maas-api-xks-ca\") pod \"bc8371e7-7efd-4728-8a3b-82ce85f361c9\" (UID: \"bc8371e7-7efd-4728-8a3b-82ce85f361c9\") " Oct 30 22:08:04.888497 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:04.888477 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8371e7-7efd-4728-8a3b-82ce85f361c9-maas-api-openshift-ca" pod "bc8371e7-7efd-4728-8a3b-82ce85f361c9" (UID: "bc8371e7-7efd-4728-8a3b-82ce85f361c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:04.888838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:04.888821 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc8371e7-7efd-4728-8a3b-82ce85f361c9-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:04.891011 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:04.890919 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc8371e7-7efd-4728-8a3b-82ce85f361c9-maas-api-xks-ca" pod "bc8371e7-7efd-4728-8a3b-82ce85f361c9" (UID: "bc8371e7-7efd-4728-8a3b-82ce85f361c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:04.900813 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:04.898030 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8371e7-7efd-4728-8a3b-82ce85f361c9-kube-api-access-d9j8n" pod "bc8371e7-7efd-4728-8a3b-82ce85f361c9" (UID: "bc8371e7-7efd-4728-8a3b-82ce85f361c9"). InnerVolumeSpecName "kube-api-access-d9j8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:04.990184 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:04.990155 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9j8n\" (UniqueName: \"kubernetes.io/projected/bc8371e7-7efd-4728-8a3b-82ce85f361c9-kube-api-access-d9j8n\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:04.990184 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:04.990183 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc8371e7-7efd-4728-8a3b-82ce85f361c9-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:05.632218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:05.632190 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc" Oct 30 22:08:05.632427 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:05.632219 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc" event={"ID":"bc8371e7-7efd-4728-8a3b-82ce85f361c9","Type":"ContainerDied","Data":"32d782290359e83f7ebe1e4844d39c8109e55e14951d53e42ed95cdfdf1a21e4"} Oct 30 22:08:05.632427 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:05.632276 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32d782290359e83f7ebe1e4844d39c8109e55e14951d53e42ed95cdfdf1a21e4" Oct 30 22:08:06.086823 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.086754 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s"] Oct 30 22:08:06.087630 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.087589 4335 cpu_manager.go:414] "containerMap: removing container" podUID="bc8371e7-7efd-4728-8a3b-82ce85f361c9" containerName="revoke-keys" Oct 30 22:08:06.087630 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.087627 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc8371e7-7efd-4728-8a3b-82ce85f361c9" containerName="revoke-keys" Oct 30 22:08:06.087818 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.087758 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc8371e7-7efd-4728-8a3b-82ce85f361c9" containerName="revoke-keys" Oct 30 22:08:06.091391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.091373 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s" Oct 30 22:08:06.150327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.150301 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s"] Oct 30 22:08:06.191956 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.191923 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:08:06.191956 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.191948 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:08:06.192762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.192742 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:06.192889 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.192874 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:08:06.206741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.206697 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0df78edc-pr26s\" (UID: \"da09a9f7-3a8c-42bb-9b45-74ecf64351d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s" Oct 30 22:08:06.206868 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.206753 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0df78edc-pr26s\" (UID: \"da09a9f7-3a8c-42bb-9b45-74ecf64351d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s" Oct 30 22:08:06.206933 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.206889 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ds6c\" (UniqueName: \"kubernetes.io/projected/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-kube-api-access-9ds6c\") pod \"maas-api-revoke-sub-0df78edc-pr26s\" (UID: \"da09a9f7-3a8c-42bb-9b45-74ecf64351d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s" Oct 30 22:08:06.308091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.308061 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0df78edc-pr26s\" (UID: \"da09a9f7-3a8c-42bb-9b45-74ecf64351d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s" Oct 30 22:08:06.308274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.308112 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0df78edc-pr26s\" (UID: \"da09a9f7-3a8c-42bb-9b45-74ecf64351d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s" Oct 30 22:08:06.308274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.308201 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ds6c\" (UniqueName: \"kubernetes.io/projected/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-kube-api-access-9ds6c\") pod \"maas-api-revoke-sub-0df78edc-pr26s\" (UID: \"da09a9f7-3a8c-42bb-9b45-74ecf64351d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s" Oct 30 22:08:06.308859 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.308831 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0df78edc-pr26s\" (UID: \"da09a9f7-3a8c-42bb-9b45-74ecf64351d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s" Oct 30 22:08:06.310926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.310898 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0df78edc-pr26s\" (UID: \"da09a9f7-3a8c-42bb-9b45-74ecf64351d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s" Oct 30 22:08:06.376812 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.376750 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ds6c\" (UniqueName: \"kubernetes.io/projected/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-kube-api-access-9ds6c\") pod \"maas-api-revoke-sub-0df78edc-pr26s\" (UID: \"da09a9f7-3a8c-42bb-9b45-74ecf64351d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s" Oct 30 22:08:06.406721 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.406698 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s" Oct 30 22:08:06.809267 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:06.809231 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s"] Oct 30 22:08:07.650164 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:07.650126 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s" event={"ID":"da09a9f7-3a8c-42bb-9b45-74ecf64351d8","Type":"ContainerStarted","Data":"3be0695f35c750aa6959979e3fffa59de18d254cd6c5877017f3b01e8e4e5a5d"} Oct 30 22:08:07.650581 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:07.650194 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s" event={"ID":"da09a9f7-3a8c-42bb-9b45-74ecf64351d8","Type":"ContainerStarted","Data":"0154cbde0c230420e84389909450dd4b801d758fe40475faef0c1abe12154886"} Oct 30 22:08:07.777920 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:07.777874 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s" podStartSLOduration=1.777860284 podStartE2EDuration="1.777860284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:07.775716175 +0000 UTC m=+1420.876147904" watchObservedRunningTime="2026-10-30 22:08:07.777860284 +0000 UTC m=+1420.878292036" Oct 30 22:08:08.660556 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:08.660521 4335 generic.go:356] "Generic (PLEG): container finished" podID="da09a9f7-3a8c-42bb-9b45-74ecf64351d8" containerID="3be0695f35c750aa6959979e3fffa59de18d254cd6c5877017f3b01e8e4e5a5d" exitCode=0 Oct 30 22:08:08.661018 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:08.660607 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s" event={"ID":"da09a9f7-3a8c-42bb-9b45-74ecf64351d8","Type":"ContainerDied","Data":"3be0695f35c750aa6959979e3fffa59de18d254cd6c5877017f3b01e8e4e5a5d"} Oct 30 22:08:09.848764 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:09.848730 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s" Oct 30 22:08:09.959636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:09.959549 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-maas-api-xks-ca\") pod \"da09a9f7-3a8c-42bb-9b45-74ecf64351d8\" (UID: \"da09a9f7-3a8c-42bb-9b45-74ecf64351d8\") " Oct 30 22:08:09.959782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:09.959653 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-kube-api-access-9ds6c\" (UniqueName: \"kubernetes.io/projected/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-kube-api-access-9ds6c\") pod \"da09a9f7-3a8c-42bb-9b45-74ecf64351d8\" (UID: \"da09a9f7-3a8c-42bb-9b45-74ecf64351d8\") " Oct 30 22:08:09.959782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:09.959748 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-maas-api-openshift-ca\") pod \"da09a9f7-3a8c-42bb-9b45-74ecf64351d8\" (UID: \"da09a9f7-3a8c-42bb-9b45-74ecf64351d8\") " Oct 30 22:08:09.960117 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:09.960096 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-maas-api-openshift-ca" pod "da09a9f7-3a8c-42bb-9b45-74ecf64351d8" (UID: "da09a9f7-3a8c-42bb-9b45-74ecf64351d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:09.960251 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:09.960178 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:09.961765 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:09.961744 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-maas-api-xks-ca" pod "da09a9f7-3a8c-42bb-9b45-74ecf64351d8" (UID: "da09a9f7-3a8c-42bb-9b45-74ecf64351d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:09.961893 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:09.961869 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-kube-api-access-9ds6c" pod "da09a9f7-3a8c-42bb-9b45-74ecf64351d8" (UID: "da09a9f7-3a8c-42bb-9b45-74ecf64351d8"). InnerVolumeSpecName "kube-api-access-9ds6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:10.061577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:10.061548 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:10.061577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:10.061578 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ds6c\" (UniqueName: \"kubernetes.io/projected/da09a9f7-3a8c-42bb-9b45-74ecf64351d8-kube-api-access-9ds6c\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:10.686821 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:10.686792 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s" Oct 30 22:08:10.686990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:10.686825 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s" event={"ID":"da09a9f7-3a8c-42bb-9b45-74ecf64351d8","Type":"ContainerDied","Data":"0154cbde0c230420e84389909450dd4b801d758fe40475faef0c1abe12154886"} Oct 30 22:08:10.686990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:10.686902 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0154cbde0c230420e84389909450dd4b801d758fe40475faef0c1abe12154886" Oct 30 22:08:15.001743 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.001712 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx"] Oct 30 22:08:15.002351 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.002337 4335 cpu_manager.go:414] "containerMap: removing container" podUID="da09a9f7-3a8c-42bb-9b45-74ecf64351d8" containerName="revoke-keys" Oct 30 22:08:15.002406 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.002352 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da09a9f7-3a8c-42bb-9b45-74ecf64351d8" containerName="revoke-keys" Oct 30 22:08:15.002458 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.002449 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="da09a9f7-3a8c-42bb-9b45-74ecf64351d8" containerName="revoke-keys" Oct 30 22:08:15.006273 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.006256 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx" Oct 30 22:08:15.064111 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.064090 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx"] Oct 30 22:08:15.106462 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.106434 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:15.106736 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.106714 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:08:15.106736 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.106714 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:08:15.107304 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.107284 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:08:15.119921 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.119900 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knlm8\" (UniqueName: \"kubernetes.io/projected/b95ec70a-e549-49b4-8160-c2ecabdba2f2-kube-api-access-knlm8\") pod \"maas-api-revoke-sub-6b56eefa-ndcbx\" (UID: \"b95ec70a-e549-49b4-8160-c2ecabdba2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx" Oct 30 22:08:15.120141 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.120122 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b95ec70a-e549-49b4-8160-c2ecabdba2f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b56eefa-ndcbx\" (UID: \"b95ec70a-e549-49b4-8160-c2ecabdba2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx" Oct 30 22:08:15.120189 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.120172 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b95ec70a-e549-49b4-8160-c2ecabdba2f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b56eefa-ndcbx\" (UID: \"b95ec70a-e549-49b4-8160-c2ecabdba2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx" Oct 30 22:08:15.221710 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.221678 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b95ec70a-e549-49b4-8160-c2ecabdba2f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b56eefa-ndcbx\" (UID: \"b95ec70a-e549-49b4-8160-c2ecabdba2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx" Oct 30 22:08:15.221834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.221765 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b95ec70a-e549-49b4-8160-c2ecabdba2f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b56eefa-ndcbx\" (UID: \"b95ec70a-e549-49b4-8160-c2ecabdba2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx" Oct 30 22:08:15.221834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.221804 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knlm8\" (UniqueName: \"kubernetes.io/projected/b95ec70a-e549-49b4-8160-c2ecabdba2f2-kube-api-access-knlm8\") pod \"maas-api-revoke-sub-6b56eefa-ndcbx\" (UID: \"b95ec70a-e549-49b4-8160-c2ecabdba2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx" Oct 30 22:08:15.222825 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.222795 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b95ec70a-e549-49b4-8160-c2ecabdba2f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b56eefa-ndcbx\" (UID: \"b95ec70a-e549-49b4-8160-c2ecabdba2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx" Oct 30 22:08:15.224691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.224666 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b95ec70a-e549-49b4-8160-c2ecabdba2f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b56eefa-ndcbx\" (UID: \"b95ec70a-e549-49b4-8160-c2ecabdba2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx" Oct 30 22:08:15.282382 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.282325 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knlm8\" (UniqueName: \"kubernetes.io/projected/b95ec70a-e549-49b4-8160-c2ecabdba2f2-kube-api-access-knlm8\") pod \"maas-api-revoke-sub-6b56eefa-ndcbx\" (UID: \"b95ec70a-e549-49b4-8160-c2ecabdba2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx" Oct 30 22:08:15.320307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.320282 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx" Oct 30 22:08:15.700876 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:08:15.700842 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95ec70a_e549_49b4_8160_c2ecabdba2f2.slice/crio-b87e2463111e34a0f29c0f6df020b031d94e34189f55d83a19f0da0bf71599f4 WatchSource:0}: Error finding container b87e2463111e34a0f29c0f6df020b031d94e34189f55d83a19f0da0bf71599f4: Status 404 returned error can't find the container with id b87e2463111e34a0f29c0f6df020b031d94e34189f55d83a19f0da0bf71599f4 Oct 30 22:08:15.702113 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.702089 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx"] Oct 30 22:08:15.736039 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:15.736017 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx" event={"ID":"b95ec70a-e549-49b4-8160-c2ecabdba2f2","Type":"ContainerStarted","Data":"b87e2463111e34a0f29c0f6df020b031d94e34189f55d83a19f0da0bf71599f4"} Oct 30 22:08:16.265820 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:16.265754 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb"] Oct 30 22:08:16.270196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:16.270174 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb" Oct 30 22:08:16.327338 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:16.327310 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb"] Oct 30 22:08:16.435731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:16.435699 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55xhj\" (UniqueName: \"kubernetes.io/projected/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-kube-api-access-55xhj\") pod \"maas-api-revoke-sub-46dd922e-9mnjb\" (UID: \"cb79fe62-f447-40a0-bce0-1472e8ef5b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb" Oct 30 22:08:16.435877 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:16.435769 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46dd922e-9mnjb\" (UID: \"cb79fe62-f447-40a0-bce0-1472e8ef5b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb" Oct 30 22:08:16.435922 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:16.435887 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46dd922e-9mnjb\" (UID: \"cb79fe62-f447-40a0-bce0-1472e8ef5b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb" Oct 30 22:08:16.537512 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:16.537436 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55xhj\" (UniqueName: \"kubernetes.io/projected/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-kube-api-access-55xhj\") pod \"maas-api-revoke-sub-46dd922e-9mnjb\" (UID: \"cb79fe62-f447-40a0-bce0-1472e8ef5b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb" Oct 30 22:08:16.537682 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:16.537506 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46dd922e-9mnjb\" (UID: \"cb79fe62-f447-40a0-bce0-1472e8ef5b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb" Oct 30 22:08:16.537682 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:16.537598 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46dd922e-9mnjb\" (UID: \"cb79fe62-f447-40a0-bce0-1472e8ef5b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb" Oct 30 22:08:16.538280 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:16.538259 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46dd922e-9mnjb\" (UID: \"cb79fe62-f447-40a0-bce0-1472e8ef5b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb" Oct 30 22:08:16.540116 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:16.540096 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46dd922e-9mnjb\" (UID: \"cb79fe62-f447-40a0-bce0-1472e8ef5b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb" Oct 30 22:08:16.602537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:16.602513 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55xhj\" (UniqueName: \"kubernetes.io/projected/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-kube-api-access-55xhj\") pod \"maas-api-revoke-sub-46dd922e-9mnjb\" (UID: \"cb79fe62-f447-40a0-bce0-1472e8ef5b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb" Oct 30 22:08:16.746267 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:16.746232 4335 generic.go:356] "Generic (PLEG): container finished" podID="b95ec70a-e549-49b4-8160-c2ecabdba2f2" containerID="4936d019327908a587624bfb942ab6e8be0259e74feb0c8d5fef68ed8187667f" exitCode=0 Oct 30 22:08:16.746408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:16.746316 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx" event={"ID":"b95ec70a-e549-49b4-8160-c2ecabdba2f2","Type":"ContainerDied","Data":"4936d019327908a587624bfb942ab6e8be0259e74feb0c8d5fef68ed8187667f"} Oct 30 22:08:16.884685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:16.884652 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb" Oct 30 22:08:17.266947 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:08:17.266913 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb79fe62_f447_40a0_bce0_1472e8ef5b6b.slice/crio-7009672ff81117fbfd09296778a9bfc2124c87ed79288801eb5a1c7818856d77 WatchSource:0}: Error finding container 7009672ff81117fbfd09296778a9bfc2124c87ed79288801eb5a1c7818856d77: Status 404 returned error can't find the container with id 7009672ff81117fbfd09296778a9bfc2124c87ed79288801eb5a1c7818856d77 Oct 30 22:08:17.268885 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:17.268863 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb"] Oct 30 22:08:17.756862 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:17.756824 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb" event={"ID":"cb79fe62-f447-40a0-bce0-1472e8ef5b6b","Type":"ContainerStarted","Data":"2c0bf7e6e920486774799aca40d09dce8fe793b1b7441a7c7a36f357d3bc5488"} Oct 30 22:08:17.757056 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:17.756910 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb" event={"ID":"cb79fe62-f447-40a0-bce0-1472e8ef5b6b","Type":"ContainerStarted","Data":"7009672ff81117fbfd09296778a9bfc2124c87ed79288801eb5a1c7818856d77"} Oct 30 22:08:17.892525 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:17.892473 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb" podStartSLOduration=1.892456505 podStartE2EDuration="1.892456505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:17.892239252 +0000 UTC m=+1430.992670985" watchObservedRunningTime="2026-10-30 22:08:17.892456505 +0000 UTC m=+1430.992888238" Oct 30 22:08:17.911287 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:17.911263 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx" Oct 30 22:08:18.058233 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:18.058197 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b95ec70a-e549-49b4-8160-c2ecabdba2f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b95ec70a-e549-49b4-8160-c2ecabdba2f2-maas-api-xks-ca\") pod \"b95ec70a-e549-49b4-8160-c2ecabdba2f2\" (UID: \"b95ec70a-e549-49b4-8160-c2ecabdba2f2\") " Oct 30 22:08:18.058433 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:18.058413 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b95ec70a-e549-49b4-8160-c2ecabdba2f2-kube-api-access-knlm8\" (UniqueName: \"kubernetes.io/projected/b95ec70a-e549-49b4-8160-c2ecabdba2f2-kube-api-access-knlm8\") pod \"b95ec70a-e549-49b4-8160-c2ecabdba2f2\" (UID: \"b95ec70a-e549-49b4-8160-c2ecabdba2f2\") " Oct 30 22:08:18.058540 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:18.058467 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b95ec70a-e549-49b4-8160-c2ecabdba2f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b95ec70a-e549-49b4-8160-c2ecabdba2f2-maas-api-openshift-ca\") pod \"b95ec70a-e549-49b4-8160-c2ecabdba2f2\" (UID: \"b95ec70a-e549-49b4-8160-c2ecabdba2f2\") " Oct 30 22:08:18.059296 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:18.059270 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b95ec70a-e549-49b4-8160-c2ecabdba2f2-maas-api-openshift-ca" pod "b95ec70a-e549-49b4-8160-c2ecabdba2f2" (UID: "b95ec70a-e549-49b4-8160-c2ecabdba2f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:18.063765 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:18.063713 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b95ec70a-e549-49b4-8160-c2ecabdba2f2-maas-api-xks-ca" pod "b95ec70a-e549-49b4-8160-c2ecabdba2f2" (UID: "b95ec70a-e549-49b4-8160-c2ecabdba2f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:18.063765 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:18.063743 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b95ec70a-e549-49b4-8160-c2ecabdba2f2-kube-api-access-knlm8" pod "b95ec70a-e549-49b4-8160-c2ecabdba2f2" (UID: "b95ec70a-e549-49b4-8160-c2ecabdba2f2"). InnerVolumeSpecName "kube-api-access-knlm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:18.160666 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:18.160635 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b95ec70a-e549-49b4-8160-c2ecabdba2f2-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:18.160666 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:18.160666 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knlm8\" (UniqueName: \"kubernetes.io/projected/b95ec70a-e549-49b4-8160-c2ecabdba2f2-kube-api-access-knlm8\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:18.160812 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:18.160681 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b95ec70a-e549-49b4-8160-c2ecabdba2f2-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:18.766850 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:18.766813 4335 generic.go:356] "Generic (PLEG): container finished" podID="cb79fe62-f447-40a0-bce0-1472e8ef5b6b" containerID="2c0bf7e6e920486774799aca40d09dce8fe793b1b7441a7c7a36f357d3bc5488" exitCode=0 Oct 30 22:08:18.767281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:18.766855 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb" event={"ID":"cb79fe62-f447-40a0-bce0-1472e8ef5b6b","Type":"ContainerDied","Data":"2c0bf7e6e920486774799aca40d09dce8fe793b1b7441a7c7a36f357d3bc5488"} Oct 30 22:08:18.768564 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:18.768536 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx" Oct 30 22:08:18.768693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:18.768582 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx" event={"ID":"b95ec70a-e549-49b4-8160-c2ecabdba2f2","Type":"ContainerDied","Data":"b87e2463111e34a0f29c0f6df020b031d94e34189f55d83a19f0da0bf71599f4"} Oct 30 22:08:18.768693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:18.768634 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b87e2463111e34a0f29c0f6df020b031d94e34189f55d83a19f0da0bf71599f4" Oct 30 22:08:19.942555 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:19.942523 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb" Oct 30 22:08:20.085996 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:20.085966 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-kube-api-access-55xhj\" (UniqueName: \"kubernetes.io/projected/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-kube-api-access-55xhj\") pod \"cb79fe62-f447-40a0-bce0-1472e8ef5b6b\" (UID: \"cb79fe62-f447-40a0-bce0-1472e8ef5b6b\") " Oct 30 22:08:20.086181 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:20.086011 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-maas-api-openshift-ca\") pod \"cb79fe62-f447-40a0-bce0-1472e8ef5b6b\" (UID: \"cb79fe62-f447-40a0-bce0-1472e8ef5b6b\") " Oct 30 22:08:20.086254 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:20.086199 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-maas-api-xks-ca\") pod \"cb79fe62-f447-40a0-bce0-1472e8ef5b6b\" (UID: \"cb79fe62-f447-40a0-bce0-1472e8ef5b6b\") " Oct 30 22:08:20.086463 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:20.086437 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-maas-api-openshift-ca" pod "cb79fe62-f447-40a0-bce0-1472e8ef5b6b" (UID: "cb79fe62-f447-40a0-bce0-1472e8ef5b6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:20.086997 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:20.086971 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:20.088519 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:20.088494 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-maas-api-xks-ca" pod "cb79fe62-f447-40a0-bce0-1472e8ef5b6b" (UID: "cb79fe62-f447-40a0-bce0-1472e8ef5b6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:20.088923 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:20.088898 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-kube-api-access-55xhj" pod "cb79fe62-f447-40a0-bce0-1472e8ef5b6b" (UID: "cb79fe62-f447-40a0-bce0-1472e8ef5b6b"). InnerVolumeSpecName "kube-api-access-55xhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:20.188039 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:20.188011 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:20.188176 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:20.188041 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55xhj\" (UniqueName: \"kubernetes.io/projected/cb79fe62-f447-40a0-bce0-1472e8ef5b6b-kube-api-access-55xhj\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:20.786574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:20.786546 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb" Oct 30 22:08:20.786789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:20.786582 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb" event={"ID":"cb79fe62-f447-40a0-bce0-1472e8ef5b6b","Type":"ContainerDied","Data":"7009672ff81117fbfd09296778a9bfc2124c87ed79288801eb5a1c7818856d77"} Oct 30 22:08:20.786789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:20.786642 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7009672ff81117fbfd09296778a9bfc2124c87ed79288801eb5a1c7818856d77" Oct 30 22:08:25.333257 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:08:25.333214 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:08:27.065030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:27.064996 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs"] Oct 30 22:08:27.100512 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:27.100486 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e683f4f1-hkfqs"] Oct 30 22:08:27.475568 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:27.475481 4335 scope.go:122] "RemoveContainer" containerID="6720af95a7d4bda5af4d5d224319560ed3374469a79ab63e56e6e7cca3f275a7" Oct 30 22:08:29.059251 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:29.059217 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1708729-a250-4d83-b4b8-1988c3630a58" path="/var/lib/kubelet/pods/d1708729-a250-4d83-b4b8-1988c3630a58/volumes" Oct 30 22:08:29.835223 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:29.835189 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:08:31.189687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.175167 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759"] Oct 30 22:08:31.189687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.176333 4335 cpu_manager.go:414] "containerMap: removing container" podUID="cb79fe62-f447-40a0-bce0-1472e8ef5b6b" containerName="revoke-keys" Oct 30 22:08:31.189687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.176356 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb79fe62-f447-40a0-bce0-1472e8ef5b6b" containerName="revoke-keys" Oct 30 22:08:31.189687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.176413 4335 cpu_manager.go:414] "containerMap: removing container" podUID="b95ec70a-e549-49b4-8160-c2ecabdba2f2" containerName="revoke-keys" Oct 30 22:08:31.189687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.176422 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b95ec70a-e549-49b4-8160-c2ecabdba2f2" containerName="revoke-keys" Oct 30 22:08:31.189687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.176590 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="b95ec70a-e549-49b4-8160-c2ecabdba2f2" containerName="revoke-keys" Oct 30 22:08:31.189687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.176633 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb79fe62-f447-40a0-bce0-1472e8ef5b6b" containerName="revoke-keys" Oct 30 22:08:31.228914 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.228885 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.244722 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.244678 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t"] Oct 30 22:08:31.389943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.326312 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-credential-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.389943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.375753 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.389943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.385838 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759"] Oct 30 22:08:31.389943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.387800 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-workload-certs\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.389943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.388570 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-data\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.389943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.388753 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-podinfo\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.389943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.388780 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istiod-ca-cert\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.389943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.388834 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-token\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.389943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.388867 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggjkq\" (UniqueName: \"kubernetes.io/projected/74ffb1a6-5089-419b-a51d-a31f5b1b8649-kube-api-access-ggjkq\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.389943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.388973 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-workload-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.389943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.388998 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-envoy\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.393587 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.393563 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t"] Oct 30 22:08:31.592026 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.591987 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-credential-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.592274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.592254 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-workload-certs\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.602636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.592398 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-data\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.602824 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.602802 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-podinfo\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.602966 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.602947 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istiod-ca-cert\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.603114 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.603097 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-data\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.625638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.603263 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-envoy\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.625843 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.625819 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-podinfo\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.625966 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.625949 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istiod-ca-cert\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.626091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.626076 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-token\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.626221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.626201 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggjkq\" (UniqueName: \"kubernetes.io/projected/74ffb1a6-5089-419b-a51d-a31f5b1b8649-kube-api-access-ggjkq\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.626343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.626327 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-workload-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.626536 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.626516 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-workload-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.626685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.626668 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-envoy\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.662780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.662748 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-credential-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.662992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.662974 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-workload-certs\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.663100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.663087 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-token\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.663214 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.663201 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cv2d\" (UniqueName: \"kubernetes.io/projected/04c45673-5196-4efe-a65c-fbf6ba5e3c08-kube-api-access-7cv2d\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.663317 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.595875 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-workload-certs\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.663497 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.629034 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-workload-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.663631 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.595638 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-credential-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.663763 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.629726 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istiod-ca-cert\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.663869 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.611947 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-data\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.677890 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.677853 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7"] Oct 30 22:08:31.681573 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.681339 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-podinfo\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.685299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.684466 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7"] Oct 30 22:08:31.685299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.684581 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.712663 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.712576 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-envoy\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.747333 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.747232 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggjkq\" (UniqueName: \"kubernetes.io/projected/74ffb1a6-5089-419b-a51d-a31f5b1b8649-kube-api-access-ggjkq\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.771948 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-credential-socket\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.772024 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istiod-ca-cert\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.772086 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-credential-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.772143 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhhp5\" (UniqueName: \"kubernetes.io/projected/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-kube-api-access-mhhp5\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.772175 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-workload-certs\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.772200 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-token\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.772224 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-workload-certs\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.772267 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cv2d\" (UniqueName: \"kubernetes.io/projected/04c45673-5196-4efe-a65c-fbf6ba5e3c08-kube-api-access-7cv2d\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.772333 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-podinfo\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.772376 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istiod-ca-cert\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.772414 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-data\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.772444 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-data\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.772485 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-token\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.772534 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-envoy\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.772562 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-podinfo\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.772593 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-envoy\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.772694 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-workload-socket\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.772756 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-workload-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.773213 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-workload-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.773508 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-credential-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.778716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.773793 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-workload-certs\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.789717 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.780790 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istiod-ca-cert\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.789717 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.781079 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-data\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.789717 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.787056 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-envoy\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.790294 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.790272 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-token\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:31.798779 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.790914 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-podinfo\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.856705 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.855167 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-token\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.866513 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.866461 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cv2d\" (UniqueName: \"kubernetes.io/projected/04c45673-5196-4efe-a65c-fbf6ba5e3c08-kube-api-access-7cv2d\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.880331 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.879406 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-envoy\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.880331 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.879450 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-podinfo\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.880331 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.879519 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-workload-socket\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.880331 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.879604 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-credential-socket\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.880331 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.880023 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-credential-socket\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.882719 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.880797 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istiod-ca-cert\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.882719 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.880904 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhhp5\" (UniqueName: \"kubernetes.io/projected/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-kube-api-access-mhhp5\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.882719 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.880942 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-workload-certs\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.882719 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.881053 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-data\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.882719 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.881096 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-token\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.882719 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.882079 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-workload-certs\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.882719 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.882152 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istiod-ca-cert\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.882719 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.882429 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-data\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.883249 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.883229 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-workload-socket\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.885175 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.885135 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-envoy\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.887022 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.887003 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-podinfo\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.930634 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.928968 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:31.945374 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.945301 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-token\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:31.946433 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:31.946371 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhhp5\" (UniqueName: \"kubernetes.io/projected/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-kube-api-access-mhhp5\") pod \"e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:32.036870 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.025407 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:32.077672 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.077644 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk"] Oct 30 22:08:32.086142 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.085777 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b92a539-pmxjk"] Oct 30 22:08:32.120867 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.120088 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:32.179021 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.178969 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp"] Oct 30 22:08:32.185238 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.185080 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.287939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.280020 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp"] Oct 30 22:08:32.326637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.325926 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-podinfo\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.337645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.331140 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-token\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.337645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.331283 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istiod-ca-cert\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.337645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.331383 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-data\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.337645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.331779 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-workload-socket\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.337645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.331860 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-credential-socket\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.337645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.331993 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-workload-certs\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.337645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.332070 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8fjp\" (UniqueName: \"kubernetes.io/projected/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-kube-api-access-h8fjp\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.337645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.332139 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-envoy\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.439299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.438263 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-podinfo\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.439299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.438314 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-token\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.439299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.438353 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istiod-ca-cert\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.439299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.438384 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-data\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.439299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.438514 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-workload-socket\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.439299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.438540 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-credential-socket\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.439299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.438585 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-workload-certs\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.439299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.438666 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8fjp\" (UniqueName: \"kubernetes.io/projected/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-kube-api-access-h8fjp\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.439299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.438698 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-envoy\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.442430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.440296 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-data\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.442430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.441262 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-credential-socket\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.442430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.441499 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-workload-socket\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.442430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.441756 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-workload-certs\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.442430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.442386 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istiod-ca-cert\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.443787 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.443763 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-envoy\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.446052 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.446031 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-podinfo\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.468752 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.468714 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t"] Oct 30 22:08:32.472311 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.472280 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:08:32.472397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.472345 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:08:32.472397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.472375 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:08:32.512456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.505820 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-token\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.512456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.506808 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8fjp\" (UniqueName: \"kubernetes.io/projected/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-kube-api-access-h8fjp\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.577565 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.577543 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:32.742344 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:08:32.742291 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice/crio-dc52df387192676062840a9952066c80ef74d1b37d55f395b1bf2bfde1da0894 WatchSource:0}: Error finding container dc52df387192676062840a9952066c80ef74d1b37d55f395b1bf2bfde1da0894: Status 404 returned error can't find the container with id dc52df387192676062840a9952066c80ef74d1b37d55f395b1bf2bfde1da0894 Oct 30 22:08:32.742695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.742675 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7"] Oct 30 22:08:32.748082 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.748048 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:08:32.748240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.748223 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:08:32.748349 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.748335 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:08:32.803326 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.803306 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759"] Oct 30 22:08:32.805835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.805525 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:08:32.805835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.805597 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:08:32.805835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:32.805653 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:08:33.003675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.003130 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" event={"ID":"87b2369a-d31f-4413-b5c6-ec4b217c3f8b","Type":"ContainerStarted","Data":"8554f5a3adb87bcc1e5fbb9a1e888c83ca7fa4290ba01ccc487f79862f978309"} Oct 30 22:08:33.003675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.003204 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" event={"ID":"87b2369a-d31f-4413-b5c6-ec4b217c3f8b","Type":"ContainerStarted","Data":"dc52df387192676062840a9952066c80ef74d1b37d55f395b1bf2bfde1da0894"} Oct 30 22:08:33.015675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.011731 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" event={"ID":"04c45673-5196-4efe-a65c-fbf6ba5e3c08","Type":"ContainerStarted","Data":"879c1f553c92b22b3d8dab8a921e5c55748ac388a69d8bc4dcec3b0ff692afa4"} Oct 30 22:08:33.015675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.011788 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" event={"ID":"04c45673-5196-4efe-a65c-fbf6ba5e3c08","Type":"ContainerStarted","Data":"20bbf9df512b85e907c9098f62402d25696f3576de39314794e11885cf1c517a"} Oct 30 22:08:33.015675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.014729 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp"] Oct 30 22:08:33.022957 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.022930 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" event={"ID":"74ffb1a6-5089-419b-a51d-a31f5b1b8649","Type":"ContainerStarted","Data":"54769dbd213f04f609b772cceb6fb162ae982cc8baabf373896c6a4a4cdaad60"} Oct 30 22:08:33.023167 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.023144 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" event={"ID":"74ffb1a6-5089-419b-a51d-a31f5b1b8649","Type":"ContainerStarted","Data":"2c3991abbdb288d38d4ed701e9ab4cce9032d3eab3eacf80acfd326c7a5ed7e2"} Oct 30 22:08:33.026815 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.026798 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:33.028282 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.028239 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:08:33.028448 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.028427 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" podUID="74ffb1a6-5089-419b-a51d-a31f5b1b8649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 22:08:33.029504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.029473 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:08:33.029788 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.029765 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:08:33.029954 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.029934 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:08:33.078873 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.078809 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7115d25-0f2d-4679-958b-d31e313b197f" path="/var/lib/kubelet/pods/f7115d25-0f2d-4679-958b-d31e313b197f/volumes" Oct 30 22:08:33.124246 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.121519 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:33.124246 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.123946 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:08:33.124246 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.124000 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" podUID="87b2369a-d31f-4413-b5c6-ec4b217c3f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 22:08:33.125437 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.125374 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" podStartSLOduration=2.125357364 podStartE2EDuration="2.125357364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:33.124912789 +0000 UTC m=+1446.225344523" watchObservedRunningTime="2026-10-30 22:08:33.125357364 +0000 UTC m=+1446.225789098" Oct 30 22:08:33.259805 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.259743 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" podStartSLOduration=2.259724443 podStartE2EDuration="2.259724443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:33.253093207 +0000 UTC m=+1446.353524940" watchObservedRunningTime="2026-10-30 22:08:33.259724443 +0000 UTC m=+1446.360156176" Oct 30 22:08:33.265109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.265083 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7"] Oct 30 22:08:33.269644 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.269604 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7" Oct 30 22:08:33.339127 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.339043 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:08:33.340191 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.339795 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:08:33.340316 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.339878 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:08:33.340414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.339921 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:33.343418 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.343398 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7"] Oct 30 22:08:33.365645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.363959 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2489bed8-5799-488d-b80b-f2e3c159e3a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ae275ca-n9xx7\" (UID: \"2489bed8-5799-488d-b80b-f2e3c159e3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7" Oct 30 22:08:33.365645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.364080 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2489bed8-5799-488d-b80b-f2e3c159e3a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ae275ca-n9xx7\" (UID: \"2489bed8-5799-488d-b80b-f2e3c159e3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7" Oct 30 22:08:33.365645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.364347 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf7f7\" (UniqueName: \"kubernetes.io/projected/2489bed8-5799-488d-b80b-f2e3c159e3a6-kube-api-access-bf7f7\") pod \"maas-api-revoke-sub-5ae275ca-n9xx7\" (UID: \"2489bed8-5799-488d-b80b-f2e3c159e3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7" Oct 30 22:08:33.378930 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.378889 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" podStartSLOduration=2.37887838 podStartE2EDuration="2.37887838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:33.374499735 +0000 UTC m=+1446.474931467" watchObservedRunningTime="2026-10-30 22:08:33.37887838 +0000 UTC m=+1446.479310111" Oct 30 22:08:33.466048 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.466013 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf7f7\" (UniqueName: \"kubernetes.io/projected/2489bed8-5799-488d-b80b-f2e3c159e3a6-kube-api-access-bf7f7\") pod \"maas-api-revoke-sub-5ae275ca-n9xx7\" (UID: \"2489bed8-5799-488d-b80b-f2e3c159e3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7" Oct 30 22:08:33.466212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.466145 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2489bed8-5799-488d-b80b-f2e3c159e3a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ae275ca-n9xx7\" (UID: \"2489bed8-5799-488d-b80b-f2e3c159e3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7" Oct 30 22:08:33.466212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.466191 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2489bed8-5799-488d-b80b-f2e3c159e3a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ae275ca-n9xx7\" (UID: \"2489bed8-5799-488d-b80b-f2e3c159e3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7" Oct 30 22:08:33.466964 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.466930 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2489bed8-5799-488d-b80b-f2e3c159e3a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ae275ca-n9xx7\" (UID: \"2489bed8-5799-488d-b80b-f2e3c159e3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7" Oct 30 22:08:33.469192 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.469161 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2489bed8-5799-488d-b80b-f2e3c159e3a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ae275ca-n9xx7\" (UID: \"2489bed8-5799-488d-b80b-f2e3c159e3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7" Oct 30 22:08:33.530298 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.529986 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf7f7\" (UniqueName: \"kubernetes.io/projected/2489bed8-5799-488d-b80b-f2e3c159e3a6-kube-api-access-bf7f7\") pod \"maas-api-revoke-sub-5ae275ca-n9xx7\" (UID: \"2489bed8-5799-488d-b80b-f2e3c159e3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7" Oct 30 22:08:33.592965 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.592869 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7" Oct 30 22:08:33.935245 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.935158 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:33.938188 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.938157 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:08:33.938403 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:33.938378 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" podUID="04c45673-5196-4efe-a65c-fbf6ba5e3c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 22:08:34.136990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:34.136794 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:08:34.136990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:34.136850 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" podUID="74ffb1a6-5089-419b-a51d-a31f5b1b8649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 22:08:34.164501 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:34.150186 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:08:34.164501 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:34.150241 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" podUID="87b2369a-d31f-4413-b5c6-ec4b217c3f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 22:08:34.465702 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:34.465670 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" event={"ID":"5c5448fc-7500-43b9-abd3-f9a8c603fbd0","Type":"ContainerStarted","Data":"21cfb229c638e174dbbcaab01daebd6afe409b2fd6c987b602f2d2592bd9c2f6"} Oct 30 22:08:34.466306 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:34.466282 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" event={"ID":"5c5448fc-7500-43b9-abd3-f9a8c603fbd0","Type":"ContainerStarted","Data":"b648a5066b1fb5e14c2fff22975b7a76046233e89dd226e13184e47695d6a6f5"} Oct 30 22:08:34.582029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:34.580539 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:34.582029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:34.581910 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:08:34.582029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:34.581960 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" podUID="5c5448fc-7500-43b9-abd3-f9a8c603fbd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 22:08:34.647794 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:34.634659 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7"] Oct 30 22:08:34.802638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:34.802237 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:34.872874 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:34.872769 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:34.905658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:34.895428 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" podStartSLOduration=2.895407208 podStartE2EDuration="2.895407208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:34.629664748 +0000 UTC m=+1447.730096481" watchObservedRunningTime="2026-10-30 22:08:34.895407208 +0000 UTC m=+1447.995838941" Oct 30 22:08:34.962641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:34.959176 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:08:34.962641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:34.959241 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" podUID="04c45673-5196-4efe-a65c-fbf6ba5e3c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 22:08:35.048928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:35.031759 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:08:35.048928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:35.031831 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" podUID="74ffb1a6-5089-419b-a51d-a31f5b1b8649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 22:08:35.140066 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:35.139917 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:08:35.140066 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:35.139999 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" podUID="87b2369a-d31f-4413-b5c6-ec4b217c3f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 22:08:35.218947 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:35.218904 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:35.260694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:35.255339 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:35.262259 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:35.261992 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:35.513775 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:35.511438 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:35.582642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:35.582550 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7" event={"ID":"2489bed8-5799-488d-b80b-f2e3c159e3a6","Type":"ContainerStarted","Data":"c04d224882fac671e0f1eb3281a5c21fcf2309ea44e808fc2cef281410c7e4f9"} Oct 30 22:08:35.582843 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:35.582734 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Oct 30 22:08:35.584013 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:35.583975 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:08:35.584239 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:35.584208 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" podUID="5c5448fc-7500-43b9-abd3-f9a8c603fbd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 22:08:35.747736 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:35.746755 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:35.890151 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:35.890109 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:35.891555 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:35.891530 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:08:35.934111 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:35.934080 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:08:35.934332 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:35.934309 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" podUID="04c45673-5196-4efe-a65c-fbf6ba5e3c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 22:08:36.017639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.014344 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:36.140694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.126755 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:08:36.140694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.126818 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" podUID="87b2369a-d31f-4413-b5c6-ec4b217c3f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 22:08:36.140694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.127357 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:08:36.140694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.127392 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" podUID="74ffb1a6-5089-419b-a51d-a31f5b1b8649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 22:08:36.257223 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.257182 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:36.263638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.257760 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:08:36.283661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.265548 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:36.286014 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.285986 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:08:36.312826 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.312797 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7"] Oct 30 22:08:36.340153 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.340118 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faf53655-gvvr7"] Oct 30 22:08:36.367479 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.366728 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:36.605640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.557432 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": context deadline exceeded" Oct 30 22:08:36.605640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.557860 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:08:36.606178 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.606109 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:08:36.606178 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.606164 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" podUID="5c5448fc-7500-43b9-abd3-f9a8c603fbd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 22:08:36.628660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.607013 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Oct 30 22:08:36.628660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.607352 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:08:36.756298 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.755780 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:36.759285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.756577 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:08:36.788556 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.788529 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:08:36.810205 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.810120 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:36.828601 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.828565 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 30 22:08:36.890529 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.890443 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:36.938294 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.933705 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:08:36.938294 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:36.933766 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" podUID="04c45673-5196-4efe-a65c-fbf6ba5e3c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 22:08:37.013782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.000056 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7" event={"ID":"2489bed8-5799-488d-b80b-f2e3c159e3a6","Type":"ContainerStarted","Data":"b7bad8af809eddf970be1e5b3d7444b2bb9ceb1a1f1fa97cb7e784f813ac6607"} Oct 30 22:08:37.016708 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.015524 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 22:08:37.016708 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.015836 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:08:37.028241 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.028211 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:08:37.032710 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.032676 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" podUID="74ffb1a6-5089-419b-a51d-a31f5b1b8649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 22:08:37.133584 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.133552 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddf30afc-138c-4265-aadd-1193e3744df2" path="/var/lib/kubelet/pods/ddf30afc-138c-4265-aadd-1193e3744df2/volumes" Oct 30 22:08:37.134445 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.134414 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:37.142772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.142743 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:08:37.142958 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.142933 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" podUID="87b2369a-d31f-4413-b5c6-ec4b217c3f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 22:08:37.162673 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.162595 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7" podStartSLOduration=4.162576379 podStartE2EDuration="4.162576379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:37.142512556 +0000 UTC m=+1450.242944290" watchObservedRunningTime="2026-10-30 22:08:37.162576379 +0000 UTC m=+1450.263008112" Oct 30 22:08:37.258754 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.258708 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:37.414695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.413663 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:37.441637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.416223 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 22:08:37.441992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.441957 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/health\": context deadline exceeded" Oct 30 22:08:37.511903 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.511873 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:08:37.568500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.568390 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:37.587509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.587477 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:08:37.587777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.587741 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" podUID="5c5448fc-7500-43b9-abd3-f9a8c603fbd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 22:08:37.612556 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.612514 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:37.871335 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.871282 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:37.898482 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:37.898436 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:38.052641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.048873 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 22:08:38.073228 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.073194 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:08:38.246697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.220020 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" podUID="04c45673-5196-4efe-a65c-fbf6ba5e3c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 22:08:38.246697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.192200 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:38.246697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.190921 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:38.246697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.220933 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 22:08:38.246697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.242870 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:08:38.246697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.242938 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" podUID="87b2369a-d31f-4413-b5c6-ec4b217c3f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 22:08:38.246697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.243079 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:08:38.246697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.243107 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" podUID="74ffb1a6-5089-419b-a51d-a31f5b1b8649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 22:08:38.306641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.297895 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:38.324102 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.324078 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:08:38.538713 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.513574 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": context deadline exceeded" Oct 30 22:08:38.601640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.578534 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:08:38.601640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.578592 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" podUID="5c5448fc-7500-43b9-abd3-f9a8c603fbd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 22:08:38.601640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.578731 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 22:08:38.640650 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.625148 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Oct 30 22:08:38.884641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.861453 4335 generic.go:356] "Generic (PLEG): container finished" podID="2489bed8-5799-488d-b80b-f2e3c159e3a6" containerID="b7bad8af809eddf970be1e5b3d7444b2bb9ceb1a1f1fa97cb7e784f813ac6607" exitCode=0 Oct 30 22:08:38.884641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.863393 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7" event={"ID":"2489bed8-5799-488d-b80b-f2e3c159e3a6","Type":"ContainerDied","Data":"b7bad8af809eddf970be1e5b3d7444b2bb9ceb1a1f1fa97cb7e784f813ac6607"} Oct 30 22:08:38.898641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.887986 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:38.898931 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.898896 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:38.929185 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.929146 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:38.934051 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.934023 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:08:38.976668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:38.975060 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" podUID="04c45673-5196-4efe-a65c-fbf6ba5e3c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 22:08:39.084647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.058822 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:39.159707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.155547 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:08:39.159707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.155605 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" podUID="74ffb1a6-5089-419b-a51d-a31f5b1b8649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 22:08:39.159707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.155923 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:08:39.159707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.155956 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" podUID="87b2369a-d31f-4413-b5c6-ec4b217c3f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 22:08:39.264112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.262724 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:39.347326 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.347290 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:39.347666 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.347639 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:39.375528 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.234916 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:39.382636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.380718 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:39.518693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.517207 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:39.585640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.582913 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:39.585640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.583255 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 22:08:39.585640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.584015 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:08:39.585640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.584054 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" podUID="5c5448fc-7500-43b9-abd3-f9a8c603fbd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 22:08:39.627785 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.627743 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:39.891429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.891385 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:39.892407 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.892386 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:08:39.920754 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.920706 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:39.930887 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.930853 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:39.948323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.948295 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:08:39.951315 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.951290 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:08:39.951514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:39.951490 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" podUID="04c45673-5196-4efe-a65c-fbf6ba5e3c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 22:08:40.048680 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.048648 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:08:40.071874 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.071835 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" podUID="74ffb1a6-5089-419b-a51d-a31f5b1b8649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 22:08:40.111649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.111449 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 22:08:40.199714 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.165344 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:08:40.199982 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.199948 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" podUID="87b2369a-d31f-4413-b5c6-ec4b217c3f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 22:08:40.200120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.178717 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:40.349692 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.348686 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:40.349692 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.349048 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:08:40.390674 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.383060 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 22:08:40.406377 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.391094 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Oct 30 22:08:40.520807 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.520704 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:40.594641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.587446 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:40.649640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.647749 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Oct 30 22:08:40.693141 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.693115 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:08:40.783741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.783669 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:08:40.811231 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.807136 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:08:40.811231 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.807210 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" podUID="5c5448fc-7500-43b9-abd3-f9a8c603fbd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 22:08:40.952732 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.948661 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:40.968936 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.966290 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:08:40.968936 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:40.966340 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" podUID="04c45673-5196-4efe-a65c-fbf6ba5e3c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 22:08:41.042737 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.041342 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:08:41.042737 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.041400 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" podUID="74ffb1a6-5089-419b-a51d-a31f5b1b8649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 22:08:41.093339 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.093304 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:41.129269 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.128498 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:41.156818 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.147739 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:08:41.156818 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.147807 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" podUID="87b2369a-d31f-4413-b5c6-ec4b217c3f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 22:08:41.162645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.159799 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7" Oct 30 22:08:41.213577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.207285 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 22:08:41.213577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.212351 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:08:41.218699 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.217125 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:08:41.240825 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.240802 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:08:41.272640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.272601 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:08:41.273299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.273283 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:08:41.275100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.275074 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2489bed8-5799-488d-b80b-f2e3c159e3a6-kube-api-access-bf7f7\" (UniqueName: \"kubernetes.io/projected/2489bed8-5799-488d-b80b-f2e3c159e3a6-kube-api-access-bf7f7\") pod \"2489bed8-5799-488d-b80b-f2e3c159e3a6\" (UID: \"2489bed8-5799-488d-b80b-f2e3c159e3a6\") " Oct 30 22:08:41.289890 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.289868 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2489bed8-5799-488d-b80b-f2e3c159e3a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2489bed8-5799-488d-b80b-f2e3c159e3a6-maas-api-xks-ca\") pod \"2489bed8-5799-488d-b80b-f2e3c159e3a6\" (UID: \"2489bed8-5799-488d-b80b-f2e3c159e3a6\") " Oct 30 22:08:41.290397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.290379 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2489bed8-5799-488d-b80b-f2e3c159e3a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2489bed8-5799-488d-b80b-f2e3c159e3a6-maas-api-openshift-ca\") pod \"2489bed8-5799-488d-b80b-f2e3c159e3a6\" (UID: \"2489bed8-5799-488d-b80b-f2e3c159e3a6\") " Oct 30 22:08:41.291880 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.290851 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2489bed8-5799-488d-b80b-f2e3c159e3a6-maas-api-openshift-ca" pod "2489bed8-5799-488d-b80b-f2e3c159e3a6" (UID: "2489bed8-5799-488d-b80b-f2e3c159e3a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:41.304696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.299538 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:08:41.317290 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.317266 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2489bed8-5799-488d-b80b-f2e3c159e3a6-kube-api-access-bf7f7" pod "2489bed8-5799-488d-b80b-f2e3c159e3a6" (UID: "2489bed8-5799-488d-b80b-f2e3c159e3a6"). InnerVolumeSpecName "kube-api-access-bf7f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:41.342296 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.323683 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 22:08:41.349217 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.349188 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:41.349536 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.349513 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2489bed8-5799-488d-b80b-f2e3c159e3a6-maas-api-xks-ca" pod "2489bed8-5799-488d-b80b-f2e3c159e3a6" (UID: "2489bed8-5799-488d-b80b-f2e3c159e3a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:41.361363 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.361344 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:08:41.395472 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.394789 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf7f7\" (UniqueName: \"kubernetes.io/projected/2489bed8-5799-488d-b80b-f2e3c159e3a6-kube-api-access-bf7f7\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:41.395472 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.394819 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2489bed8-5799-488d-b80b-f2e3c159e3a6-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:41.395472 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.394837 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2489bed8-5799-488d-b80b-f2e3c159e3a6-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:41.403183 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.403161 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:08:41.459435 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.459409 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:08:41.579680 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.579571 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:08:41.579924 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:41.579898 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" podUID="5c5448fc-7500-43b9-abd3-f9a8c603fbd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 22:08:42.031704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:42.031276 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:08:42.031704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:42.031339 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" podUID="74ffb1a6-5089-419b-a51d-a31f5b1b8649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 22:08:42.112898 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:42.112864 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7" Oct 30 22:08:42.117666 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:42.116228 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7" event={"ID":"2489bed8-5799-488d-b80b-f2e3c159e3a6","Type":"ContainerDied","Data":"c04d224882fac671e0f1eb3281a5c21fcf2309ea44e808fc2cef281410c7e4f9"} Oct 30 22:08:42.121638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:42.117882 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c04d224882fac671e0f1eb3281a5c21fcf2309ea44e808fc2cef281410c7e4f9" Oct 30 22:08:42.204646 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:42.204537 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:42.207438 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:42.206189 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:42.207438 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:42.207296 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:08:42.586923 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:42.586888 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:08:42.587216 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:42.587184 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" podUID="5c5448fc-7500-43b9-abd3-f9a8c603fbd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 22:08:42.827751 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:42.827720 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:42.828296 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:42.828274 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:42.829540 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:42.829520 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:08:43.030377 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:43.030300 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:43.030748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:43.030728 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:43.031454 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:43.031434 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:08:43.062571 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:43.062544 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn"] Oct 30 22:08:43.062950 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:43.062583 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57b29051-562mn"] Oct 30 22:08:43.583101 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:43.583071 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:43.583531 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:43.583515 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:43.584219 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:43.584205 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:08:45.062025 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:45.061980 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1faa26c2-e6db-41c6-8598-a296df151cea" path="/var/lib/kubelet/pods/1faa26c2-e6db-41c6-8598-a296df151cea/volumes" Oct 30 22:08:47.952089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:47.952056 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2"] Oct 30 22:08:47.952746 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:47.952722 4335 cpu_manager.go:414] "containerMap: removing container" podUID="2489bed8-5799-488d-b80b-f2e3c159e3a6" containerName="revoke-keys" Oct 30 22:08:47.952804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:47.952748 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2489bed8-5799-488d-b80b-f2e3c159e3a6" containerName="revoke-keys" Oct 30 22:08:47.952866 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:47.952856 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="2489bed8-5799-488d-b80b-f2e3c159e3a6" containerName="revoke-keys" Oct 30 22:08:47.956872 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:47.956850 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2" Oct 30 22:08:48.015572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.015544 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2"] Oct 30 22:08:48.047212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.047180 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce46877e-24e6-4ac1-8207-cb23b169f3f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e55b4d99-b4bx2\" (UID: \"ce46877e-24e6-4ac1-8207-cb23b169f3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2" Oct 30 22:08:48.047429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.047410 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdnpp\" (UniqueName: \"kubernetes.io/projected/ce46877e-24e6-4ac1-8207-cb23b169f3f4-kube-api-access-fdnpp\") pod \"maas-api-revoke-sub-e55b4d99-b4bx2\" (UID: \"ce46877e-24e6-4ac1-8207-cb23b169f3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2" Oct 30 22:08:48.047538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.047519 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce46877e-24e6-4ac1-8207-cb23b169f3f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e55b4d99-b4bx2\" (UID: \"ce46877e-24e6-4ac1-8207-cb23b169f3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2" Oct 30 22:08:48.056810 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.056782 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:48.056939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.056825 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:08:48.056939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.056832 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:08:48.057064 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.056971 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:08:48.149350 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.149323 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdnpp\" (UniqueName: \"kubernetes.io/projected/ce46877e-24e6-4ac1-8207-cb23b169f3f4-kube-api-access-fdnpp\") pod \"maas-api-revoke-sub-e55b4d99-b4bx2\" (UID: \"ce46877e-24e6-4ac1-8207-cb23b169f3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2" Oct 30 22:08:48.149490 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.149370 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce46877e-24e6-4ac1-8207-cb23b169f3f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e55b4d99-b4bx2\" (UID: \"ce46877e-24e6-4ac1-8207-cb23b169f3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2" Oct 30 22:08:48.149566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.149547 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce46877e-24e6-4ac1-8207-cb23b169f3f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e55b4d99-b4bx2\" (UID: \"ce46877e-24e6-4ac1-8207-cb23b169f3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2" Oct 30 22:08:48.150047 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.150022 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce46877e-24e6-4ac1-8207-cb23b169f3f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e55b4d99-b4bx2\" (UID: \"ce46877e-24e6-4ac1-8207-cb23b169f3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2" Oct 30 22:08:48.152096 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.152080 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce46877e-24e6-4ac1-8207-cb23b169f3f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e55b4d99-b4bx2\" (UID: \"ce46877e-24e6-4ac1-8207-cb23b169f3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2" Oct 30 22:08:48.210236 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.210164 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdnpp\" (UniqueName: \"kubernetes.io/projected/ce46877e-24e6-4ac1-8207-cb23b169f3f4-kube-api-access-fdnpp\") pod \"maas-api-revoke-sub-e55b4d99-b4bx2\" (UID: \"ce46877e-24e6-4ac1-8207-cb23b169f3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2" Oct 30 22:08:48.271251 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.271224 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2" Oct 30 22:08:48.576649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.576603 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9"] Oct 30 22:08:48.582036 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.582017 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9" Oct 30 22:08:48.643656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.643602 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9"] Oct 30 22:08:48.662081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.662056 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6xzc\" (UniqueName: \"kubernetes.io/projected/dbd17d98-9598-4da0-a37a-1f260a977070-kube-api-access-k6xzc\") pod \"maas-api-revoke-sub-59fe371e-jknw9\" (UID: \"dbd17d98-9598-4da0-a37a-1f260a977070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9" Oct 30 22:08:48.662189 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.662109 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd17d98-9598-4da0-a37a-1f260a977070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59fe371e-jknw9\" (UID: \"dbd17d98-9598-4da0-a37a-1f260a977070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9" Oct 30 22:08:48.662267 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.662244 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd17d98-9598-4da0-a37a-1f260a977070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59fe371e-jknw9\" (UID: \"dbd17d98-9598-4da0-a37a-1f260a977070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9" Oct 30 22:08:48.689007 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:08:48.688977 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce46877e_24e6_4ac1_8207_cb23b169f3f4.slice/crio-33bf373d50dccc023dc835766dda5fef4d99ee98c06875fcd3b01a8aa8a55da8 WatchSource:0}: Error finding container 33bf373d50dccc023dc835766dda5fef4d99ee98c06875fcd3b01a8aa8a55da8: Status 404 returned error can't find the container with id 33bf373d50dccc023dc835766dda5fef4d99ee98c06875fcd3b01a8aa8a55da8 Oct 30 22:08:48.690756 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.690733 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2"] Oct 30 22:08:48.763477 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.763453 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6xzc\" (UniqueName: \"kubernetes.io/projected/dbd17d98-9598-4da0-a37a-1f260a977070-kube-api-access-k6xzc\") pod \"maas-api-revoke-sub-59fe371e-jknw9\" (UID: \"dbd17d98-9598-4da0-a37a-1f260a977070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9" Oct 30 22:08:48.763604 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.763505 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd17d98-9598-4da0-a37a-1f260a977070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59fe371e-jknw9\" (UID: \"dbd17d98-9598-4da0-a37a-1f260a977070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9" Oct 30 22:08:48.763604 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.763528 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd17d98-9598-4da0-a37a-1f260a977070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59fe371e-jknw9\" (UID: \"dbd17d98-9598-4da0-a37a-1f260a977070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9" Oct 30 22:08:48.764079 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.764059 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd17d98-9598-4da0-a37a-1f260a977070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59fe371e-jknw9\" (UID: \"dbd17d98-9598-4da0-a37a-1f260a977070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9" Oct 30 22:08:48.765900 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.765879 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd17d98-9598-4da0-a37a-1f260a977070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59fe371e-jknw9\" (UID: \"dbd17d98-9598-4da0-a37a-1f260a977070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9" Oct 30 22:08:48.824038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.824007 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6xzc\" (UniqueName: \"kubernetes.io/projected/dbd17d98-9598-4da0-a37a-1f260a977070-kube-api-access-k6xzc\") pod \"maas-api-revoke-sub-59fe371e-jknw9\" (UID: \"dbd17d98-9598-4da0-a37a-1f260a977070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9" Oct 30 22:08:48.897281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:48.897212 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9" Oct 30 22:08:49.059174 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:49.059146 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58"] Oct 30 22:08:49.059174 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:49.059180 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62119c3d-tjp58"] Oct 30 22:08:49.153052 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:49.152976 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2" event={"ID":"ce46877e-24e6-4ac1-8207-cb23b169f3f4","Type":"ContainerStarted","Data":"e77d20d818bf9c9dac255d83d804b0c4ae2178e69ad51e68ddf14cf014ced1ba"} Oct 30 22:08:49.153052 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:49.153035 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2" event={"ID":"ce46877e-24e6-4ac1-8207-cb23b169f3f4","Type":"ContainerStarted","Data":"33bf373d50dccc023dc835766dda5fef4d99ee98c06875fcd3b01a8aa8a55da8"} Oct 30 22:08:49.273130 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:49.273075 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2" podStartSLOduration=2.273057723 podStartE2EDuration="2.273057723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:49.267583447 +0000 UTC m=+1462.368015177" watchObservedRunningTime="2026-10-30 22:08:49.273057723 +0000 UTC m=+1462.373489456" Oct 30 22:08:49.285697 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:08:49.285668 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd17d98_9598_4da0_a37a_1f260a977070.slice/crio-80d926c5b9eb9a2ee8ad76c007cfe15691a9c987fb9fc81d47ab3581ea69c719 WatchSource:0}: Error finding container 80d926c5b9eb9a2ee8ad76c007cfe15691a9c987fb9fc81d47ab3581ea69c719: Status 404 returned error can't find the container with id 80d926c5b9eb9a2ee8ad76c007cfe15691a9c987fb9fc81d47ab3581ea69c719 Oct 30 22:08:49.287058 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:49.287037 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9"] Oct 30 22:08:50.165691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:50.165661 4335 generic.go:356] "Generic (PLEG): container finished" podID="ce46877e-24e6-4ac1-8207-cb23b169f3f4" containerID="e77d20d818bf9c9dac255d83d804b0c4ae2178e69ad51e68ddf14cf014ced1ba" exitCode=0 Oct 30 22:08:50.166105 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:50.165747 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2" event={"ID":"ce46877e-24e6-4ac1-8207-cb23b169f3f4","Type":"ContainerDied","Data":"e77d20d818bf9c9dac255d83d804b0c4ae2178e69ad51e68ddf14cf014ced1ba"} Oct 30 22:08:50.167685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:50.167662 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9" event={"ID":"dbd17d98-9598-4da0-a37a-1f260a977070","Type":"ContainerStarted","Data":"d1ae9c89b672171b0e0729e1bf52d6758911fa825e5c3396b96cce86608c5780"} Oct 30 22:08:50.167791 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:50.167707 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9" event={"ID":"dbd17d98-9598-4da0-a37a-1f260a977070","Type":"ContainerStarted","Data":"80d926c5b9eb9a2ee8ad76c007cfe15691a9c987fb9fc81d47ab3581ea69c719"} Oct 30 22:08:51.060870 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.060838 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddaab237-ddfc-4371-8288-49752bd1529e" path="/var/lib/kubelet/pods/ddaab237-ddfc-4371-8288-49752bd1529e/volumes" Oct 30 22:08:51.177263 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.177224 4335 generic.go:356] "Generic (PLEG): container finished" podID="dbd17d98-9598-4da0-a37a-1f260a977070" containerID="d1ae9c89b672171b0e0729e1bf52d6758911fa825e5c3396b96cce86608c5780" exitCode=0 Oct 30 22:08:51.177704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.177287 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9" event={"ID":"dbd17d98-9598-4da0-a37a-1f260a977070","Type":"ContainerDied","Data":"d1ae9c89b672171b0e0729e1bf52d6758911fa825e5c3396b96cce86608c5780"} Oct 30 22:08:51.369514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.364421 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2" Oct 30 22:08:51.396701 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.396551 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9" Oct 30 22:08:51.498028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.497995 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce46877e-24e6-4ac1-8207-cb23b169f3f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce46877e-24e6-4ac1-8207-cb23b169f3f4-maas-api-xks-ca\") pod \"ce46877e-24e6-4ac1-8207-cb23b169f3f4\" (UID: \"ce46877e-24e6-4ac1-8207-cb23b169f3f4\") " Oct 30 22:08:51.498170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.498087 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce46877e-24e6-4ac1-8207-cb23b169f3f4-kube-api-access-fdnpp\" (UniqueName: \"kubernetes.io/projected/ce46877e-24e6-4ac1-8207-cb23b169f3f4-kube-api-access-fdnpp\") pod \"ce46877e-24e6-4ac1-8207-cb23b169f3f4\" (UID: \"ce46877e-24e6-4ac1-8207-cb23b169f3f4\") " Oct 30 22:08:51.498659 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.498633 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd17d98-9598-4da0-a37a-1f260a977070-kube-api-access-k6xzc\" (UniqueName: \"kubernetes.io/projected/dbd17d98-9598-4da0-a37a-1f260a977070-kube-api-access-k6xzc\") pod \"dbd17d98-9598-4da0-a37a-1f260a977070\" (UID: \"dbd17d98-9598-4da0-a37a-1f260a977070\") " Oct 30 22:08:51.498936 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.498916 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce46877e-24e6-4ac1-8207-cb23b169f3f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce46877e-24e6-4ac1-8207-cb23b169f3f4-maas-api-openshift-ca\") pod \"ce46877e-24e6-4ac1-8207-cb23b169f3f4\" (UID: \"ce46877e-24e6-4ac1-8207-cb23b169f3f4\") " Oct 30 22:08:51.499501 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.499409 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce46877e-24e6-4ac1-8207-cb23b169f3f4-maas-api-openshift-ca" pod "ce46877e-24e6-4ac1-8207-cb23b169f3f4" (UID: "ce46877e-24e6-4ac1-8207-cb23b169f3f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:51.499501 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.499466 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd17d98-9598-4da0-a37a-1f260a977070-maas-api-openshift-ca" pod "dbd17d98-9598-4da0-a37a-1f260a977070" (UID: "dbd17d98-9598-4da0-a37a-1f260a977070"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:51.499713 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.499104 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd17d98-9598-4da0-a37a-1f260a977070-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd17d98-9598-4da0-a37a-1f260a977070-maas-api-openshift-ca\") pod \"dbd17d98-9598-4da0-a37a-1f260a977070\" (UID: \"dbd17d98-9598-4da0-a37a-1f260a977070\") " Oct 30 22:08:51.499932 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.499913 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbd17d98-9598-4da0-a37a-1f260a977070-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd17d98-9598-4da0-a37a-1f260a977070-maas-api-xks-ca\") pod \"dbd17d98-9598-4da0-a37a-1f260a977070\" (UID: \"dbd17d98-9598-4da0-a37a-1f260a977070\") " Oct 30 22:08:51.500927 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.500898 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce46877e-24e6-4ac1-8207-cb23b169f3f4-kube-api-access-fdnpp" pod "ce46877e-24e6-4ac1-8207-cb23b169f3f4" (UID: "ce46877e-24e6-4ac1-8207-cb23b169f3f4"). InnerVolumeSpecName "kube-api-access-fdnpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:51.502007 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.501156 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce46877e-24e6-4ac1-8207-cb23b169f3f4-maas-api-xks-ca" pod "ce46877e-24e6-4ac1-8207-cb23b169f3f4" (UID: "ce46877e-24e6-4ac1-8207-cb23b169f3f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:51.502007 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.501835 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd17d98-9598-4da0-a37a-1f260a977070-kube-api-access-k6xzc" pod "dbd17d98-9598-4da0-a37a-1f260a977070" (UID: "dbd17d98-9598-4da0-a37a-1f260a977070"). InnerVolumeSpecName "kube-api-access-k6xzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:51.502397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.502370 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6xzc\" (UniqueName: \"kubernetes.io/projected/dbd17d98-9598-4da0-a37a-1f260a977070-kube-api-access-k6xzc\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:51.502502 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.502404 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce46877e-24e6-4ac1-8207-cb23b169f3f4-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:51.502502 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.502420 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd17d98-9598-4da0-a37a-1f260a977070-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:51.502502 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.502435 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce46877e-24e6-4ac1-8207-cb23b169f3f4-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:51.502502 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.502448 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdnpp\" (UniqueName: \"kubernetes.io/projected/ce46877e-24e6-4ac1-8207-cb23b169f3f4-kube-api-access-fdnpp\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:51.509998 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.509974 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbd17d98-9598-4da0-a37a-1f260a977070-maas-api-xks-ca" pod "dbd17d98-9598-4da0-a37a-1f260a977070" (UID: "dbd17d98-9598-4da0-a37a-1f260a977070"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:51.553012 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.552986 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8"] Oct 30 22:08:51.553678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.553664 4335 cpu_manager.go:414] "containerMap: removing container" podUID="dbd17d98-9598-4da0-a37a-1f260a977070" containerName="revoke-keys" Oct 30 22:08:51.553735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.553680 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd17d98-9598-4da0-a37a-1f260a977070" containerName="revoke-keys" Oct 30 22:08:51.553735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.553698 4335 cpu_manager.go:414] "containerMap: removing container" podUID="ce46877e-24e6-4ac1-8207-cb23b169f3f4" containerName="revoke-keys" Oct 30 22:08:51.553735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.553704 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce46877e-24e6-4ac1-8207-cb23b169f3f4" containerName="revoke-keys" Oct 30 22:08:51.553834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.553815 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd17d98-9598-4da0-a37a-1f260a977070" containerName="revoke-keys" Oct 30 22:08:51.553834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.553826 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce46877e-24e6-4ac1-8207-cb23b169f3f4" containerName="revoke-keys" Oct 30 22:08:51.557658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.557640 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8" Oct 30 22:08:51.603878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.603816 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd17d98-9598-4da0-a37a-1f260a977070-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:51.618986 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.618963 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8"] Oct 30 22:08:51.704906 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.704871 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e699390-82de-4cc7-9eb2-14f627713ca0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-273e4030-8mwd8\" (UID: \"4e699390-82de-4cc7-9eb2-14f627713ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8" Oct 30 22:08:51.705052 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.704927 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64cj9\" (UniqueName: \"kubernetes.io/projected/4e699390-82de-4cc7-9eb2-14f627713ca0-kube-api-access-64cj9\") pod \"maas-api-revoke-sub-273e4030-8mwd8\" (UID: \"4e699390-82de-4cc7-9eb2-14f627713ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8" Oct 30 22:08:51.705127 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.705102 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e699390-82de-4cc7-9eb2-14f627713ca0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-273e4030-8mwd8\" (UID: \"4e699390-82de-4cc7-9eb2-14f627713ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8" Oct 30 22:08:51.806671 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.806634 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e699390-82de-4cc7-9eb2-14f627713ca0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-273e4030-8mwd8\" (UID: \"4e699390-82de-4cc7-9eb2-14f627713ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8" Oct 30 22:08:51.806874 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.806689 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64cj9\" (UniqueName: \"kubernetes.io/projected/4e699390-82de-4cc7-9eb2-14f627713ca0-kube-api-access-64cj9\") pod \"maas-api-revoke-sub-273e4030-8mwd8\" (UID: \"4e699390-82de-4cc7-9eb2-14f627713ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8" Oct 30 22:08:51.806874 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.806767 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e699390-82de-4cc7-9eb2-14f627713ca0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-273e4030-8mwd8\" (UID: \"4e699390-82de-4cc7-9eb2-14f627713ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8" Oct 30 22:08:51.807406 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.807375 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e699390-82de-4cc7-9eb2-14f627713ca0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-273e4030-8mwd8\" (UID: \"4e699390-82de-4cc7-9eb2-14f627713ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8" Oct 30 22:08:51.809462 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.809435 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e699390-82de-4cc7-9eb2-14f627713ca0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-273e4030-8mwd8\" (UID: \"4e699390-82de-4cc7-9eb2-14f627713ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8" Oct 30 22:08:51.867753 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.867681 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64cj9\" (UniqueName: \"kubernetes.io/projected/4e699390-82de-4cc7-9eb2-14f627713ca0-kube-api-access-64cj9\") pod \"maas-api-revoke-sub-273e4030-8mwd8\" (UID: \"4e699390-82de-4cc7-9eb2-14f627713ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8" Oct 30 22:08:51.871530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:51.871512 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8" Oct 30 22:08:52.187846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:52.187781 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9" Oct 30 22:08:52.187846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:52.187824 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9" event={"ID":"dbd17d98-9598-4da0-a37a-1f260a977070","Type":"ContainerDied","Data":"80d926c5b9eb9a2ee8ad76c007cfe15691a9c987fb9fc81d47ab3581ea69c719"} Oct 30 22:08:52.188258 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:52.187871 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80d926c5b9eb9a2ee8ad76c007cfe15691a9c987fb9fc81d47ab3581ea69c719" Oct 30 22:08:52.189459 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:52.189441 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2" Oct 30 22:08:52.189569 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:52.189482 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2" event={"ID":"ce46877e-24e6-4ac1-8207-cb23b169f3f4","Type":"ContainerDied","Data":"33bf373d50dccc023dc835766dda5fef4d99ee98c06875fcd3b01a8aa8a55da8"} Oct 30 22:08:52.189569 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:52.189521 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33bf373d50dccc023dc835766dda5fef4d99ee98c06875fcd3b01a8aa8a55da8" Oct 30 22:08:52.259208 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:08:52.259181 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e699390_82de_4cc7_9eb2_14f627713ca0.slice/crio-1171a5496c15523a432bcf9a6f9050784f8c9f5b73ed29cbf489f628ab6cd09a WatchSource:0}: Error finding container 1171a5496c15523a432bcf9a6f9050784f8c9f5b73ed29cbf489f628ab6cd09a: Status 404 returned error can't find the container with id 1171a5496c15523a432bcf9a6f9050784f8c9f5b73ed29cbf489f628ab6cd09a Oct 30 22:08:52.260185 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:52.260164 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8"] Oct 30 22:08:53.200650 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:53.200596 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8" event={"ID":"4e699390-82de-4cc7-9eb2-14f627713ca0","Type":"ContainerStarted","Data":"bf662f801f22f2a918dfe1970495730e346ef6b57ffc5299406ccf3bc5db131d"} Oct 30 22:08:53.201028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:53.200675 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8" event={"ID":"4e699390-82de-4cc7-9eb2-14f627713ca0","Type":"ContainerStarted","Data":"1171a5496c15523a432bcf9a6f9050784f8c9f5b73ed29cbf489f628ab6cd09a"} Oct 30 22:08:53.320210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:53.320166 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8" podStartSLOduration=2.320152136 podStartE2EDuration="2.320152136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:53.315224424 +0000 UTC m=+1466.415656157" watchObservedRunningTime="2026-10-30 22:08:53.320152136 +0000 UTC m=+1466.420583868" Oct 30 22:08:54.316677 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.316265 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf"] Oct 30 22:08:54.332670 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.324979 4335 generic.go:356] "Generic (PLEG): container finished" podID="4e699390-82de-4cc7-9eb2-14f627713ca0" containerID="bf662f801f22f2a918dfe1970495730e346ef6b57ffc5299406ccf3bc5db131d" exitCode=0 Oct 30 22:08:54.332670 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.329743 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8" event={"ID":"4e699390-82de-4cc7-9eb2-14f627713ca0","Type":"ContainerDied","Data":"bf662f801f22f2a918dfe1970495730e346ef6b57ffc5299406ccf3bc5db131d"} Oct 30 22:08:54.332670 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.329857 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.350888 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.350364 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf"] Oct 30 22:08:54.439277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.439247 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a7b06b\"/\"distinct-2-w5-a7b06b-kserve-self-signed-certs\"" Oct 30 22:08:54.455342 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.455308 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-dshm\") pod \"distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.455503 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.455354 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-model-cache\") pod \"distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.455503 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.455447 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-tmp-dir\") pod \"distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.455698 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.455585 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9tnv\" (UniqueName: \"kubernetes.io/projected/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-kube-api-access-g9tnv\") pod \"distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.455698 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.455667 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-home\") pod \"distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.455813 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.455737 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-tls-certs\") pod \"distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.556685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.556643 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-dshm\") pod \"distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.556871 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.556692 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-model-cache\") pod \"distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.556871 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.556718 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-tmp-dir\") pod \"distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.556871 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.556810 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9tnv\" (UniqueName: \"kubernetes.io/projected/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-kube-api-access-g9tnv\") pod \"distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.556871 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.556850 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-home\") pod \"distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.557099 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.556895 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-tls-certs\") pod \"distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.557150 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.557107 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-model-cache\") pod \"distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.557384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.557365 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-tmp-dir\") pod \"distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.557503 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.557480 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-home\") pod \"distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.559148 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.559123 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-dshm\") pod \"distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.559555 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.559540 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-tls-certs\") pod \"distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.624682 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.623921 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9tnv\" (UniqueName: \"kubernetes.io/projected/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-kube-api-access-g9tnv\") pod \"distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:54.673646 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:54.673228 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:55.341666 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:08:55.332263 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4bee3da_5d5b_4663_9f31_bf0b2388e06e.slice/crio-17433e435513640b9da61af3e43f759c579bfa829903af95a373e4bf92f73327 WatchSource:0}: Error finding container 17433e435513640b9da61af3e43f759c579bfa829903af95a373e4bf92f73327: Status 404 returned error can't find the container with id 17433e435513640b9da61af3e43f759c579bfa829903af95a373e4bf92f73327 Oct 30 22:08:55.500794 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:55.500766 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf"] Oct 30 22:08:55.501039 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:55.501015 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" event={"ID":"a4bee3da-5d5b-4663-9f31-bf0b2388e06e","Type":"ContainerStarted","Data":"17433e435513640b9da61af3e43f759c579bfa829903af95a373e4bf92f73327"} Oct 30 22:08:55.634479 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:08:55.629261 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:08:56.376052 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:56.376020 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m"] Oct 30 22:08:56.419349 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:56.419306 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:56.496309 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:56.496257 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 22:08:56.631506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:56.631403 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Oct 30 22:08:56.764297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:56.764254 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 22:08:56.819685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:56.819649 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:56.820460 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:56.820438 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:56.866605 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:56.866567 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m"] Oct 30 22:08:56.911893 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:56.910644 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 30 22:08:56.981642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:56.968104 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2rm5\" (UniqueName: \"kubernetes.io/projected/769e08d0-bf5e-404e-8b7a-adf43dcd0905-kube-api-access-r2rm5\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:56.981642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:56.968156 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-workload-certs\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:56.981642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:56.968187 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-data\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:56.981642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:56.968294 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istiod-ca-cert\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:56.981642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:56.968334 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-podinfo\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:56.981642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:56.968417 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-envoy\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:56.981642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:56.968468 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-token\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:56.981642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:56.968515 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-credential-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:56.981642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:56.968568 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-workload-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.015909 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.015881 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8" Oct 30 22:08:57.070232 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.070178 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2rm5\" (UniqueName: \"kubernetes.io/projected/769e08d0-bf5e-404e-8b7a-adf43dcd0905-kube-api-access-r2rm5\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.071790 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.071762 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-workload-certs\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.071992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.071975 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-data\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.072284 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.072264 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istiod-ca-cert\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.072443 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.072427 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-podinfo\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.072671 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.072651 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-envoy\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.072845 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.072827 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-token\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.073010 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.072992 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-credential-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.077555 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.073838 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istiod-ca-cert\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.077555 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.074102 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-workload-certs\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.077555 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.074329 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-data\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.077555 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.075251 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-credential-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.085208 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.084300 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-workload-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.085208 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.084701 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-workload-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.091721 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.091700 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-podinfo\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.092349 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.092330 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-envoy\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.158434 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.158360 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2rm5\" (UniqueName: \"kubernetes.io/projected/769e08d0-bf5e-404e-8b7a-adf43dcd0905-kube-api-access-r2rm5\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.185804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.185770 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e699390-82de-4cc7-9eb2-14f627713ca0-kube-api-access-64cj9\" (UniqueName: \"kubernetes.io/projected/4e699390-82de-4cc7-9eb2-14f627713ca0-kube-api-access-64cj9\") pod \"4e699390-82de-4cc7-9eb2-14f627713ca0\" (UID: \"4e699390-82de-4cc7-9eb2-14f627713ca0\") " Oct 30 22:08:57.186276 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.186253 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e699390-82de-4cc7-9eb2-14f627713ca0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e699390-82de-4cc7-9eb2-14f627713ca0-maas-api-xks-ca\") pod \"4e699390-82de-4cc7-9eb2-14f627713ca0\" (UID: \"4e699390-82de-4cc7-9eb2-14f627713ca0\") " Oct 30 22:08:57.186529 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.186510 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e699390-82de-4cc7-9eb2-14f627713ca0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e699390-82de-4cc7-9eb2-14f627713ca0-maas-api-openshift-ca\") pod \"4e699390-82de-4cc7-9eb2-14f627713ca0\" (UID: \"4e699390-82de-4cc7-9eb2-14f627713ca0\") " Oct 30 22:08:57.188189 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.188121 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e699390-82de-4cc7-9eb2-14f627713ca0-maas-api-openshift-ca" pod "4e699390-82de-4cc7-9eb2-14f627713ca0" (UID: "4e699390-82de-4cc7-9eb2-14f627713ca0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:57.190752 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.190716 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e699390-82de-4cc7-9eb2-14f627713ca0-kube-api-access-64cj9" pod "4e699390-82de-4cc7-9eb2-14f627713ca0" (UID: "4e699390-82de-4cc7-9eb2-14f627713ca0"). InnerVolumeSpecName "kube-api-access-64cj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:57.195152 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.195125 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-token\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.211971 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.211928 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e699390-82de-4cc7-9eb2-14f627713ca0-maas-api-xks-ca" pod "4e699390-82de-4cc7-9eb2-14f627713ca0" (UID: "4e699390-82de-4cc7-9eb2-14f627713ca0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:57.294703 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.290023 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64cj9\" (UniqueName: \"kubernetes.io/projected/4e699390-82de-4cc7-9eb2-14f627713ca0-kube-api-access-64cj9\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:57.294703 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.290057 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e699390-82de-4cc7-9eb2-14f627713ca0-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:57.294703 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.290074 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e699390-82de-4cc7-9eb2-14f627713ca0-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:57.467093 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.467002 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:57.863495 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.863464 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8" Oct 30 22:08:57.863704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.863499 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8" event={"ID":"4e699390-82de-4cc7-9eb2-14f627713ca0","Type":"ContainerDied","Data":"1171a5496c15523a432bcf9a6f9050784f8c9f5b73ed29cbf489f628ab6cd09a"} Oct 30 22:08:57.863704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.863550 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1171a5496c15523a432bcf9a6f9050784f8c9f5b73ed29cbf489f628ab6cd09a" Oct 30 22:08:57.865459 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.865433 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" event={"ID":"a4bee3da-5d5b-4663-9f31-bf0b2388e06e","Type":"ContainerStarted","Data":"c0b4fd4aba13d721c0a9c6bdcf4c332776507182489fa495b764117a78901a87"} Oct 30 22:08:57.921776 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:08:57.921752 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769e08d0_bf5e_404e_8b7a_adf43dcd0905.slice/crio-d49efbdbf9f32a662e25e90304cbac4e31895c779a879beff5ff29a8eab133d4 WatchSource:0}: Error finding container d49efbdbf9f32a662e25e90304cbac4e31895c779a879beff5ff29a8eab133d4: Status 404 returned error can't find the container with id d49efbdbf9f32a662e25e90304cbac4e31895c779a879beff5ff29a8eab133d4 Oct 30 22:08:57.922470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.922449 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m"] Oct 30 22:08:57.924513 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.924485 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:08:57.924604 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.924544 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:08:57.924604 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.924573 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:08:57.990516 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:57.990473 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podStartSLOduration=3.990460547 podStartE2EDuration="3.990460547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:57.984398209 +0000 UTC m=+1471.084829940" watchObservedRunningTime="2026-10-30 22:08:57.990460547 +0000 UTC m=+1471.090892320" Oct 30 22:08:58.674471 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:58.673577 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:08:58.876336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:58.876300 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" event={"ID":"769e08d0-bf5e-404e-8b7a-adf43dcd0905","Type":"ContainerStarted","Data":"18ba68e9205228fe0bf3ac3622860c1cd4bbd08d20f61b64ff632720e97009e1"} Oct 30 22:08:58.876547 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:58.876379 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" event={"ID":"769e08d0-bf5e-404e-8b7a-adf43dcd0905","Type":"ContainerStarted","Data":"d49efbdbf9f32a662e25e90304cbac4e31895c779a879beff5ff29a8eab133d4"} Oct 30 22:08:59.133875 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.133721 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" podStartSLOduration=4.133698086 podStartE2EDuration="4.133698086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:59.048520644 +0000 UTC m=+1472.148952375" watchObservedRunningTime="2026-10-30 22:08:59.133698086 +0000 UTC m=+1472.234129820" Oct 30 22:08:59.351875 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.351844 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx"] Oct 30 22:08:59.352520 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.352507 4335 cpu_manager.go:414] "containerMap: removing container" podUID="4e699390-82de-4cc7-9eb2-14f627713ca0" containerName="revoke-keys" Oct 30 22:08:59.352562 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.352522 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e699390-82de-4cc7-9eb2-14f627713ca0" containerName="revoke-keys" Oct 30 22:08:59.352659 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.352648 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e699390-82de-4cc7-9eb2-14f627713ca0" containerName="revoke-keys" Oct 30 22:08:59.356414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.356398 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx" Oct 30 22:08:59.413686 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.413099 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:59.422736 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.415912 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:08:59.422736 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.416344 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:08:59.422736 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.420696 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:08:59.450650 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.444222 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx"] Oct 30 22:08:59.467790 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.467767 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:08:59.468873 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.468847 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:08:59.468988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.468903 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" podUID="769e08d0-bf5e-404e-8b7a-adf43dcd0905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 22:08:59.542585 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.542546 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb2f107-516a-4fbc-8d31-512df067bba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-700438b0-7jdwx\" (UID: \"3fb2f107-516a-4fbc-8d31-512df067bba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx" Oct 30 22:08:59.542852 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.542827 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s494n\" (UniqueName: \"kubernetes.io/projected/3fb2f107-516a-4fbc-8d31-512df067bba1-kube-api-access-s494n\") pod \"maas-api-revoke-sub-700438b0-7jdwx\" (UID: \"3fb2f107-516a-4fbc-8d31-512df067bba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx" Oct 30 22:08:59.542954 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.542934 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb2f107-516a-4fbc-8d31-512df067bba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-700438b0-7jdwx\" (UID: \"3fb2f107-516a-4fbc-8d31-512df067bba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx" Oct 30 22:08:59.644834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.644799 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s494n\" (UniqueName: \"kubernetes.io/projected/3fb2f107-516a-4fbc-8d31-512df067bba1-kube-api-access-s494n\") pod \"maas-api-revoke-sub-700438b0-7jdwx\" (UID: \"3fb2f107-516a-4fbc-8d31-512df067bba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx" Oct 30 22:08:59.645017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.644883 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb2f107-516a-4fbc-8d31-512df067bba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-700438b0-7jdwx\" (UID: \"3fb2f107-516a-4fbc-8d31-512df067bba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx" Oct 30 22:08:59.645118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.645029 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb2f107-516a-4fbc-8d31-512df067bba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-700438b0-7jdwx\" (UID: \"3fb2f107-516a-4fbc-8d31-512df067bba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx" Oct 30 22:08:59.646020 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.645884 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb2f107-516a-4fbc-8d31-512df067bba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-700438b0-7jdwx\" (UID: \"3fb2f107-516a-4fbc-8d31-512df067bba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx" Oct 30 22:08:59.647862 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.647839 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb2f107-516a-4fbc-8d31-512df067bba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-700438b0-7jdwx\" (UID: \"3fb2f107-516a-4fbc-8d31-512df067bba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx" Oct 30 22:08:59.717277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.717206 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s494n\" (UniqueName: \"kubernetes.io/projected/3fb2f107-516a-4fbc-8d31-512df067bba1-kube-api-access-s494n\") pod \"maas-api-revoke-sub-700438b0-7jdwx\" (UID: \"3fb2f107-516a-4fbc-8d31-512df067bba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx" Oct 30 22:08:59.969871 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:08:59.969803 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx" Oct 30 22:09:00.430680 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:00.428599 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx"] Oct 30 22:09:00.467555 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:00.467528 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:09:00.467690 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:00.467576 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" podUID="769e08d0-bf5e-404e-8b7a-adf43dcd0905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 22:09:00.926640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:00.924724 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx" event={"ID":"3fb2f107-516a-4fbc-8d31-512df067bba1","Type":"ContainerStarted","Data":"bec126e17e7bbd5dff8a84eb17ccec27b3c659fde28856054b678d82f84fdddb"} Oct 30 22:09:00.926640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:00.924793 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx" event={"ID":"3fb2f107-516a-4fbc-8d31-512df067bba1","Type":"ContainerStarted","Data":"e0d37754582b743dd628eaf774d4b2bf8c2e4f046f8f3a19348f705b3d67c200"} Oct 30 22:09:01.040527 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:01.040468 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx" podStartSLOduration=2.040455173 podStartE2EDuration="2.040455173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:01.038778074 +0000 UTC m=+1474.139209806" watchObservedRunningTime="2026-10-30 22:09:01.040455173 +0000 UTC m=+1474.140886948" Oct 30 22:09:01.467470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:01.467429 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:09:01.467704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:01.467514 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" podUID="769e08d0-bf5e-404e-8b7a-adf43dcd0905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 22:09:01.939526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:01.939491 4335 generic.go:356] "Generic (PLEG): container finished" podID="3fb2f107-516a-4fbc-8d31-512df067bba1" containerID="bec126e17e7bbd5dff8a84eb17ccec27b3c659fde28856054b678d82f84fdddb" exitCode=0 Oct 30 22:09:01.940037 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:01.939552 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx" event={"ID":"3fb2f107-516a-4fbc-8d31-512df067bba1","Type":"ContainerDied","Data":"bec126e17e7bbd5dff8a84eb17ccec27b3c659fde28856054b678d82f84fdddb"} Oct 30 22:09:02.468304 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.468263 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:09:02.468477 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.468327 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" podUID="769e08d0-bf5e-404e-8b7a-adf43dcd0905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 22:09:02.729247 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.729147 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2"] Oct 30 22:09:02.734531 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.734505 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.791732 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.791706 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-istio-data\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.791885 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.791751 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a060b45e-4975-47da-ace0-889ca76b3424-istiod-ca-cert\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.791951 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.791877 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-credential-socket\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.792004 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.791994 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-istio-envoy\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.792070 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.792041 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a060b45e-4975-47da-ace0-889ca76b3424-istio-podinfo\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.792126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.792082 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-workload-socket\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.792177 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.792145 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-workload-certs\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.792225 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.792189 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a060b45e-4975-47da-ace0-889ca76b3424-istio-token\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.792262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.792222 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85v8m\" (UniqueName: \"kubernetes.io/projected/a060b45e-4975-47da-ace0-889ca76b3424-kube-api-access-85v8m\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.855009 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.854968 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2"] Oct 30 22:09:02.893634 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.893588 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-istio-envoy\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.893799 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.893653 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a060b45e-4975-47da-ace0-889ca76b3424-istio-podinfo\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.893799 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.893695 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-workload-socket\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.893799 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.893761 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-workload-certs\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.893957 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.893809 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a060b45e-4975-47da-ace0-889ca76b3424-istio-token\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.893957 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.893835 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85v8m\" (UniqueName: \"kubernetes.io/projected/a060b45e-4975-47da-ace0-889ca76b3424-kube-api-access-85v8m\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.894063 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.893999 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-istio-data\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.894111 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.894057 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a060b45e-4975-47da-ace0-889ca76b3424-istiod-ca-cert\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.894160 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.894113 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-workload-socket\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.894160 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.894141 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-workload-certs\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.894255 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.894125 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-credential-socket\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.894377 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.894355 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-istio-data\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.894461 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.894441 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-credential-socket\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.894760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.894742 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a060b45e-4975-47da-ace0-889ca76b3424-istiod-ca-cert\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.936140 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.921152 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a060b45e-4975-47da-ace0-889ca76b3424-istio-podinfo\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.936140 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.926830 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-istio-envoy\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.955056 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.955031 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a060b45e-4975-47da-ace0-889ca76b3424-istio-token\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:02.955604 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:02.955567 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85v8m\" (UniqueName: \"kubernetes.io/projected/a060b45e-4975-47da-ace0-889ca76b3424-kube-api-access-85v8m\") pod \"e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:03.053650 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.053214 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:03.063642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.062866 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj"] Oct 30 22:09:03.071751 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.070781 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d0818d-rpstj"] Oct 30 22:09:03.162045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.162019 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx" Oct 30 22:09:03.243291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.241726 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb2f107-516a-4fbc-8d31-512df067bba1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb2f107-516a-4fbc-8d31-512df067bba1-maas-api-openshift-ca\") pod \"3fb2f107-516a-4fbc-8d31-512df067bba1\" (UID: \"3fb2f107-516a-4fbc-8d31-512df067bba1\") " Oct 30 22:09:03.243291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.241865 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fb2f107-516a-4fbc-8d31-512df067bba1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb2f107-516a-4fbc-8d31-512df067bba1-maas-api-xks-ca\") pod \"3fb2f107-516a-4fbc-8d31-512df067bba1\" (UID: \"3fb2f107-516a-4fbc-8d31-512df067bba1\") " Oct 30 22:09:03.243291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.241968 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb2f107-516a-4fbc-8d31-512df067bba1-kube-api-access-s494n\" (UniqueName: \"kubernetes.io/projected/3fb2f107-516a-4fbc-8d31-512df067bba1-kube-api-access-s494n\") pod \"3fb2f107-516a-4fbc-8d31-512df067bba1\" (UID: \"3fb2f107-516a-4fbc-8d31-512df067bba1\") " Oct 30 22:09:03.243291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.242589 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb2f107-516a-4fbc-8d31-512df067bba1-maas-api-openshift-ca" pod "3fb2f107-516a-4fbc-8d31-512df067bba1" (UID: "3fb2f107-516a-4fbc-8d31-512df067bba1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:03.247665 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.246964 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fb2f107-516a-4fbc-8d31-512df067bba1-maas-api-xks-ca" pod "3fb2f107-516a-4fbc-8d31-512df067bba1" (UID: "3fb2f107-516a-4fbc-8d31-512df067bba1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:03.248033 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.248009 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb2f107-516a-4fbc-8d31-512df067bba1-kube-api-access-s494n" pod "3fb2f107-516a-4fbc-8d31-512df067bba1" (UID: "3fb2f107-516a-4fbc-8d31-512df067bba1"). InnerVolumeSpecName "kube-api-access-s494n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:03.351483 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.351451 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb2f107-516a-4fbc-8d31-512df067bba1-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:03.351739 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.351721 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s494n\" (UniqueName: \"kubernetes.io/projected/3fb2f107-516a-4fbc-8d31-512df067bba1-kube-api-access-s494n\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:03.351832 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.351821 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb2f107-516a-4fbc-8d31-512df067bba1-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:03.468104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.468071 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:09:03.468272 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.468123 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" podUID="769e08d0-bf5e-404e-8b7a-adf43dcd0905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 22:09:03.577985 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:09:03.577949 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda060b45e_4975_47da_ace0_889ca76b3424.slice/crio-c88564d2ff3a2431614c965aa4ee64265b27fb13b47adf4d67961bd0fa5f11ce WatchSource:0}: Error finding container c88564d2ff3a2431614c965aa4ee64265b27fb13b47adf4d67961bd0fa5f11ce: Status 404 returned error can't find the container with id c88564d2ff3a2431614c965aa4ee64265b27fb13b47adf4d67961bd0fa5f11ce Oct 30 22:09:03.578186 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.578164 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2"] Oct 30 22:09:03.580518 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.580489 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:09:03.580594 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.580550 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:09:03.580668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.580597 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:09:03.836571 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.836537 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn"] Oct 30 22:09:03.837879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.837855 4335 cpu_manager.go:414] "containerMap: removing container" podUID="3fb2f107-516a-4fbc-8d31-512df067bba1" containerName="revoke-keys" Oct 30 22:09:03.837991 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.837880 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb2f107-516a-4fbc-8d31-512df067bba1" containerName="revoke-keys" Oct 30 22:09:03.838138 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.838124 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb2f107-516a-4fbc-8d31-512df067bba1" containerName="revoke-keys" Oct 30 22:09:03.843849 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.842997 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn" Oct 30 22:09:03.888116 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.888056 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn"] Oct 30 22:09:03.962067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.961337 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75828d3a-9c23-4319-bed3-eee5091cbf6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-929a9e57-jkgxn\" (UID: \"75828d3a-9c23-4319-bed3-eee5091cbf6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn" Oct 30 22:09:03.962067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.961458 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s85v\" (UniqueName: \"kubernetes.io/projected/75828d3a-9c23-4319-bed3-eee5091cbf6d-kube-api-access-7s85v\") pod \"maas-api-revoke-sub-929a9e57-jkgxn\" (UID: \"75828d3a-9c23-4319-bed3-eee5091cbf6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn" Oct 30 22:09:03.962067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.961654 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75828d3a-9c23-4319-bed3-eee5091cbf6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-929a9e57-jkgxn\" (UID: \"75828d3a-9c23-4319-bed3-eee5091cbf6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn" Oct 30 22:09:03.974295 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.973445 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" event={"ID":"a060b45e-4975-47da-ace0-889ca76b3424","Type":"ContainerStarted","Data":"1842a7f6fbef16976866fa7ddb6aec796ae616013f4a474b4bf53c916c531ee4"} Oct 30 22:09:03.974295 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.973510 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" event={"ID":"a060b45e-4975-47da-ace0-889ca76b3424","Type":"ContainerStarted","Data":"c88564d2ff3a2431614c965aa4ee64265b27fb13b47adf4d67961bd0fa5f11ce"} Oct 30 22:09:03.979786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.979549 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx" event={"ID":"3fb2f107-516a-4fbc-8d31-512df067bba1","Type":"ContainerDied","Data":"e0d37754582b743dd628eaf774d4b2bf8c2e4f046f8f3a19348f705b3d67c200"} Oct 30 22:09:03.979786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.979603 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0d37754582b743dd628eaf774d4b2bf8c2e4f046f8f3a19348f705b3d67c200" Oct 30 22:09:03.979786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:03.979734 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx" Oct 30 22:09:04.055241 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:04.055211 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:04.056834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:04.056793 4335 patch_prober.go:29] interesting pod/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:09:04.056981 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:04.056852 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" podUID="a060b45e-4975-47da-ace0-889ca76b3424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 22:09:04.063242 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:04.063213 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75828d3a-9c23-4319-bed3-eee5091cbf6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-929a9e57-jkgxn\" (UID: \"75828d3a-9c23-4319-bed3-eee5091cbf6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn" Oct 30 22:09:04.063397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:04.063377 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s85v\" (UniqueName: \"kubernetes.io/projected/75828d3a-9c23-4319-bed3-eee5091cbf6d-kube-api-access-7s85v\") pod \"maas-api-revoke-sub-929a9e57-jkgxn\" (UID: \"75828d3a-9c23-4319-bed3-eee5091cbf6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn" Oct 30 22:09:04.063761 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:04.063502 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75828d3a-9c23-4319-bed3-eee5091cbf6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-929a9e57-jkgxn\" (UID: \"75828d3a-9c23-4319-bed3-eee5091cbf6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn" Oct 30 22:09:04.064427 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:04.064394 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75828d3a-9c23-4319-bed3-eee5091cbf6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-929a9e57-jkgxn\" (UID: \"75828d3a-9c23-4319-bed3-eee5091cbf6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn" Oct 30 22:09:04.068020 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:04.067998 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75828d3a-9c23-4319-bed3-eee5091cbf6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-929a9e57-jkgxn\" (UID: \"75828d3a-9c23-4319-bed3-eee5091cbf6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn" Oct 30 22:09:04.125235 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:04.125197 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s85v\" (UniqueName: \"kubernetes.io/projected/75828d3a-9c23-4319-bed3-eee5091cbf6d-kube-api-access-7s85v\") pod \"maas-api-revoke-sub-929a9e57-jkgxn\" (UID: \"75828d3a-9c23-4319-bed3-eee5091cbf6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn" Oct 30 22:09:04.160286 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:04.160180 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn" Oct 30 22:09:04.201744 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:04.201679 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" podStartSLOduration=2.201658696 podStartE2EDuration="2.201658696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:04.190566044 +0000 UTC m=+1477.290997801" watchObservedRunningTime="2026-10-30 22:09:04.201658696 +0000 UTC m=+1477.302090429" Oct 30 22:09:04.468085 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:04.468008 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:09:04.468217 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:04.468084 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" podUID="769e08d0-bf5e-404e-8b7a-adf43dcd0905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 22:09:04.573828 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:09:04.573746 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75828d3a_9c23_4319_bed3_eee5091cbf6d.slice/crio-c9880381d383f1d84f262052ec8a3c39a7e6af8e7b962f2f3acd5a0d93c01eb6 WatchSource:0}: Error finding container c9880381d383f1d84f262052ec8a3c39a7e6af8e7b962f2f3acd5a0d93c01eb6: Status 404 returned error can't find the container with id c9880381d383f1d84f262052ec8a3c39a7e6af8e7b962f2f3acd5a0d93c01eb6 Oct 30 22:09:04.577500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:04.577477 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn"] Oct 30 22:09:04.674396 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:04.674337 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:09:04.692666 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:04.692636 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:09:04.992508 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:04.992470 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn" event={"ID":"75828d3a-9c23-4319-bed3-eee5091cbf6d","Type":"ContainerStarted","Data":"c9880381d383f1d84f262052ec8a3c39a7e6af8e7b962f2f3acd5a0d93c01eb6"} Oct 30 22:09:05.048968 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:05.048941 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:09:05.054980 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:05.054954 4335 patch_prober.go:29] interesting pod/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:09:05.055107 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:05.055002 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" podUID="a060b45e-4975-47da-ace0-889ca76b3424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 22:09:05.069735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:05.069700 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c82efd63-6d58-424c-bcff-772354c678f9" path="/var/lib/kubelet/pods/c82efd63-6d58-424c-bcff-772354c678f9/volumes" Oct 30 22:09:05.783246 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:05.783218 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:09:05.783875 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:05.783844 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:09:05.784945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:05.784926 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:09:06.036288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:06.036197 4335 generic.go:356] "Generic (PLEG): container finished" podID="75828d3a-9c23-4319-bed3-eee5091cbf6d" containerID="2e2f393fe9398a42ab888425a9bf33b915ab1b45351595390459d31541442476" exitCode=0 Oct 30 22:09:06.037442 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:06.037416 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn" event={"ID":"75828d3a-9c23-4319-bed3-eee5091cbf6d","Type":"ContainerDied","Data":"2e2f393fe9398a42ab888425a9bf33b915ab1b45351595390459d31541442476"} Oct 30 22:09:06.055881 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:06.055853 4335 patch_prober.go:29] interesting pod/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:09:06.056031 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:06.055907 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" podUID="a060b45e-4975-47da-ace0-889ca76b3424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 22:09:07.055817 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:07.055784 4335 patch_prober.go:29] interesting pod/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:09:07.056237 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:07.055855 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" podUID="a060b45e-4975-47da-ace0-889ca76b3424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 22:09:07.262100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:07.262076 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn" Oct 30 22:09:07.335878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:07.335844 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75828d3a-9c23-4319-bed3-eee5091cbf6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75828d3a-9c23-4319-bed3-eee5091cbf6d-maas-api-xks-ca\") pod \"75828d3a-9c23-4319-bed3-eee5091cbf6d\" (UID: \"75828d3a-9c23-4319-bed3-eee5091cbf6d\") " Oct 30 22:09:07.336025 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:07.335935 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75828d3a-9c23-4319-bed3-eee5091cbf6d-kube-api-access-7s85v\" (UniqueName: \"kubernetes.io/projected/75828d3a-9c23-4319-bed3-eee5091cbf6d-kube-api-access-7s85v\") pod \"75828d3a-9c23-4319-bed3-eee5091cbf6d\" (UID: \"75828d3a-9c23-4319-bed3-eee5091cbf6d\") " Oct 30 22:09:07.336025 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:07.335981 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75828d3a-9c23-4319-bed3-eee5091cbf6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75828d3a-9c23-4319-bed3-eee5091cbf6d-maas-api-openshift-ca\") pod \"75828d3a-9c23-4319-bed3-eee5091cbf6d\" (UID: \"75828d3a-9c23-4319-bed3-eee5091cbf6d\") " Oct 30 22:09:07.336900 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:07.336875 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75828d3a-9c23-4319-bed3-eee5091cbf6d-maas-api-openshift-ca" pod "75828d3a-9c23-4319-bed3-eee5091cbf6d" (UID: "75828d3a-9c23-4319-bed3-eee5091cbf6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:07.339361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:07.339270 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75828d3a-9c23-4319-bed3-eee5091cbf6d-maas-api-xks-ca" pod "75828d3a-9c23-4319-bed3-eee5091cbf6d" (UID: "75828d3a-9c23-4319-bed3-eee5091cbf6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:07.340349 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:07.340326 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75828d3a-9c23-4319-bed3-eee5091cbf6d-kube-api-access-7s85v" pod "75828d3a-9c23-4319-bed3-eee5091cbf6d" (UID: "75828d3a-9c23-4319-bed3-eee5091cbf6d"). InnerVolumeSpecName "kube-api-access-7s85v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:07.438328 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:07.438297 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75828d3a-9c23-4319-bed3-eee5091cbf6d-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:07.438328 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:07.438322 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s85v\" (UniqueName: \"kubernetes.io/projected/75828d3a-9c23-4319-bed3-eee5091cbf6d-kube-api-access-7s85v\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:07.438328 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:07.438332 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75828d3a-9c23-4319-bed3-eee5091cbf6d-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:08.055089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.055051 4335 patch_prober.go:29] interesting pod/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:09:08.055294 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.055112 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" podUID="a060b45e-4975-47da-ace0-889ca76b3424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 22:09:08.055294 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.055259 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn" Oct 30 22:09:08.055428 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.055258 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn" event={"ID":"75828d3a-9c23-4319-bed3-eee5091cbf6d","Type":"ContainerDied","Data":"c9880381d383f1d84f262052ec8a3c39a7e6af8e7b962f2f3acd5a0d93c01eb6"} Oct 30 22:09:08.055428 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.055378 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9880381d383f1d84f262052ec8a3c39a7e6af8e7b962f2f3acd5a0d93c01eb6" Oct 30 22:09:08.524553 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.524473 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj"] Oct 30 22:09:08.525590 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.525561 4335 cpu_manager.go:414] "containerMap: removing container" podUID="75828d3a-9c23-4319-bed3-eee5091cbf6d" containerName="revoke-keys" Oct 30 22:09:08.525590 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.525589 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75828d3a-9c23-4319-bed3-eee5091cbf6d" containerName="revoke-keys" Oct 30 22:09:08.525869 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.525851 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="75828d3a-9c23-4319-bed3-eee5091cbf6d" containerName="revoke-keys" Oct 30 22:09:08.529814 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.529783 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj" Oct 30 22:09:08.555468 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.555443 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b5c2b3-a188-48ea-9352-9cb449f19335-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29d8c9d7-r4hcj\" (UID: \"d6b5c2b3-a188-48ea-9352-9cb449f19335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj" Oct 30 22:09:08.555544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.555487 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqqrk\" (UniqueName: \"kubernetes.io/projected/d6b5c2b3-a188-48ea-9352-9cb449f19335-kube-api-access-sqqrk\") pod \"maas-api-revoke-sub-29d8c9d7-r4hcj\" (UID: \"d6b5c2b3-a188-48ea-9352-9cb449f19335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj" Oct 30 22:09:08.555683 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.555663 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b5c2b3-a188-48ea-9352-9cb449f19335-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29d8c9d7-r4hcj\" (UID: \"d6b5c2b3-a188-48ea-9352-9cb449f19335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj" Oct 30 22:09:08.572459 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.572432 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj"] Oct 30 22:09:08.583943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.583920 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:08.584016 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.583992 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:08.620845 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.620819 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:09:08.621008 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.620842 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:08.657013 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.656985 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b5c2b3-a188-48ea-9352-9cb449f19335-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29d8c9d7-r4hcj\" (UID: \"d6b5c2b3-a188-48ea-9352-9cb449f19335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj" Oct 30 22:09:08.657131 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.657036 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqqrk\" (UniqueName: \"kubernetes.io/projected/d6b5c2b3-a188-48ea-9352-9cb449f19335-kube-api-access-sqqrk\") pod \"maas-api-revoke-sub-29d8c9d7-r4hcj\" (UID: \"d6b5c2b3-a188-48ea-9352-9cb449f19335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj" Oct 30 22:09:08.657174 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.657161 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b5c2b3-a188-48ea-9352-9cb449f19335-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29d8c9d7-r4hcj\" (UID: \"d6b5c2b3-a188-48ea-9352-9cb449f19335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj" Oct 30 22:09:08.657892 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.657856 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b5c2b3-a188-48ea-9352-9cb449f19335-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29d8c9d7-r4hcj\" (UID: \"d6b5c2b3-a188-48ea-9352-9cb449f19335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj" Oct 30 22:09:08.659593 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.659569 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b5c2b3-a188-48ea-9352-9cb449f19335-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29d8c9d7-r4hcj\" (UID: \"d6b5c2b3-a188-48ea-9352-9cb449f19335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj" Oct 30 22:09:08.726323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.726292 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqqrk\" (UniqueName: \"kubernetes.io/projected/d6b5c2b3-a188-48ea-9352-9cb449f19335-kube-api-access-sqqrk\") pod \"maas-api-revoke-sub-29d8c9d7-r4hcj\" (UID: \"d6b5c2b3-a188-48ea-9352-9cb449f19335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj" Oct 30 22:09:08.844944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:08.844911 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj" Oct 30 22:09:09.055252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:09.055218 4335 patch_prober.go:29] interesting pod/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:09:09.055440 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:09.055273 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" podUID="a060b45e-4975-47da-ace0-889ca76b3424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 22:09:09.257210 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:09:09.257183 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b5c2b3_a188_48ea_9352_9cb449f19335.slice/crio-3b7164653fd54f3e012599b354782062ffc317036a594d862fd830f7469a3a28 WatchSource:0}: Error finding container 3b7164653fd54f3e012599b354782062ffc317036a594d862fd830f7469a3a28: Status 404 returned error can't find the container with id 3b7164653fd54f3e012599b354782062ffc317036a594d862fd830f7469a3a28 Oct 30 22:09:09.258179 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:09.258149 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj"] Oct 30 22:09:10.055257 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.055221 4335 patch_prober.go:29] interesting pod/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:09:10.055655 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.055255 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn"] Oct 30 22:09:10.055655 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.055279 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" podUID="a060b45e-4975-47da-ace0-889ca76b3424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 22:09:10.076334 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.076306 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj" event={"ID":"d6b5c2b3-a188-48ea-9352-9cb449f19335","Type":"ContainerStarted","Data":"7583103ed647c6e278a9d87ee469120a1e61df8992945cfc7c0729451c3402e9"} Oct 30 22:09:10.076470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.076358 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj" event={"ID":"d6b5c2b3-a188-48ea-9352-9cb449f19335","Type":"ContainerStarted","Data":"3b7164653fd54f3e012599b354782062ffc317036a594d862fd830f7469a3a28"} Oct 30 22:09:10.102840 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.102814 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ec3c884-7ggzn"] Oct 30 22:09:10.193317 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.193273 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj" podStartSLOduration=2.193261211 podStartE2EDuration="2.193261211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:10.191220716 +0000 UTC m=+1483.291652449" watchObservedRunningTime="2026-10-30 22:09:10.193261211 +0000 UTC m=+1483.293692942" Oct 30 22:09:10.487558 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.487483 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws"] Oct 30 22:09:10.521058 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.521024 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws" Oct 30 22:09:10.547440 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.547416 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws"] Oct 30 22:09:10.585681 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.585653 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8797e804-75ad-40d8-87c8-dcb6152e9af8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d944301f-bbrws\" (UID: \"8797e804-75ad-40d8-87c8-dcb6152e9af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws" Oct 30 22:09:10.585881 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.585859 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8797e804-75ad-40d8-87c8-dcb6152e9af8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d944301f-bbrws\" (UID: \"8797e804-75ad-40d8-87c8-dcb6152e9af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws" Oct 30 22:09:10.585985 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.585969 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp8nw\" (UniqueName: \"kubernetes.io/projected/8797e804-75ad-40d8-87c8-dcb6152e9af8-kube-api-access-cp8nw\") pod \"maas-api-revoke-sub-d944301f-bbrws\" (UID: \"8797e804-75ad-40d8-87c8-dcb6152e9af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws" Oct 30 22:09:10.687485 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.687448 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8797e804-75ad-40d8-87c8-dcb6152e9af8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d944301f-bbrws\" (UID: \"8797e804-75ad-40d8-87c8-dcb6152e9af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws" Oct 30 22:09:10.687701 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.687635 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8797e804-75ad-40d8-87c8-dcb6152e9af8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d944301f-bbrws\" (UID: \"8797e804-75ad-40d8-87c8-dcb6152e9af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws" Oct 30 22:09:10.687768 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.687752 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp8nw\" (UniqueName: \"kubernetes.io/projected/8797e804-75ad-40d8-87c8-dcb6152e9af8-kube-api-access-cp8nw\") pod \"maas-api-revoke-sub-d944301f-bbrws\" (UID: \"8797e804-75ad-40d8-87c8-dcb6152e9af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws" Oct 30 22:09:10.689049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.689023 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8797e804-75ad-40d8-87c8-dcb6152e9af8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d944301f-bbrws\" (UID: \"8797e804-75ad-40d8-87c8-dcb6152e9af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws" Oct 30 22:09:10.690577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.690556 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8797e804-75ad-40d8-87c8-dcb6152e9af8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d944301f-bbrws\" (UID: \"8797e804-75ad-40d8-87c8-dcb6152e9af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws" Oct 30 22:09:10.748581 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.748519 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp8nw\" (UniqueName: \"kubernetes.io/projected/8797e804-75ad-40d8-87c8-dcb6152e9af8-kube-api-access-cp8nw\") pod \"maas-api-revoke-sub-d944301f-bbrws\" (UID: \"8797e804-75ad-40d8-87c8-dcb6152e9af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws" Oct 30 22:09:10.844858 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:10.844829 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws" Oct 30 22:09:11.063786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:11.063744 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b2d9a7-fc83-452d-a515-3e579dd3b79f" path="/var/lib/kubelet/pods/34b2d9a7-fc83-452d-a515-3e579dd3b79f/volumes" Oct 30 22:09:11.095290 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:11.095252 4335 generic.go:356] "Generic (PLEG): container finished" podID="d6b5c2b3-a188-48ea-9352-9cb449f19335" containerID="7583103ed647c6e278a9d87ee469120a1e61df8992945cfc7c0729451c3402e9" exitCode=0 Oct 30 22:09:11.095648 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:11.095553 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj" event={"ID":"d6b5c2b3-a188-48ea-9352-9cb449f19335","Type":"ContainerDied","Data":"7583103ed647c6e278a9d87ee469120a1e61df8992945cfc7c0729451c3402e9"} Oct 30 22:09:11.168144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:11.168109 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:11.168784 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:11.168466 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:11.169484 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:11.169466 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:09:11.240696 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:09:11.238075 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8797e804_75ad_40d8_87c8_dcb6152e9af8.slice/crio-0cee531c8e5732f231ee25a4d75f4f3ce77a279c26a6ebd829ead72348ae2523 WatchSource:0}: Error finding container 0cee531c8e5732f231ee25a4d75f4f3ce77a279c26a6ebd829ead72348ae2523: Status 404 returned error can't find the container with id 0cee531c8e5732f231ee25a4d75f4f3ce77a279c26a6ebd829ead72348ae2523 Oct 30 22:09:11.275832 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:11.275806 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws"] Oct 30 22:09:12.109759 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:12.109728 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws" event={"ID":"8797e804-75ad-40d8-87c8-dcb6152e9af8","Type":"ContainerStarted","Data":"0a9d187d683d679e6aff08a10dfbbe907325d061481068887e3b95ef6c1cb0ad"} Oct 30 22:09:12.110364 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:12.109780 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws" event={"ID":"8797e804-75ad-40d8-87c8-dcb6152e9af8","Type":"ContainerStarted","Data":"0cee531c8e5732f231ee25a4d75f4f3ce77a279c26a6ebd829ead72348ae2523"} Oct 30 22:09:12.232704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:12.232547 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws" podStartSLOduration=2.232530262 podStartE2EDuration="2.232530262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:12.224170601 +0000 UTC m=+1485.324602340" watchObservedRunningTime="2026-10-30 22:09:12.232530262 +0000 UTC m=+1485.332961997" Oct 30 22:09:12.285253 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:12.285230 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj" Oct 30 22:09:12.411180 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:12.411072 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b5c2b3-a188-48ea-9352-9cb449f19335-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b5c2b3-a188-48ea-9352-9cb449f19335-maas-api-openshift-ca\") pod \"d6b5c2b3-a188-48ea-9352-9cb449f19335\" (UID: \"d6b5c2b3-a188-48ea-9352-9cb449f19335\") " Oct 30 22:09:12.411770 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:12.411739 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b5c2b3-a188-48ea-9352-9cb449f19335-kube-api-access-sqqrk\" (UniqueName: \"kubernetes.io/projected/d6b5c2b3-a188-48ea-9352-9cb449f19335-kube-api-access-sqqrk\") pod \"d6b5c2b3-a188-48ea-9352-9cb449f19335\" (UID: \"d6b5c2b3-a188-48ea-9352-9cb449f19335\") " Oct 30 22:09:12.411902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:12.411790 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6b5c2b3-a188-48ea-9352-9cb449f19335-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b5c2b3-a188-48ea-9352-9cb449f19335-maas-api-xks-ca\") pod \"d6b5c2b3-a188-48ea-9352-9cb449f19335\" (UID: \"d6b5c2b3-a188-48ea-9352-9cb449f19335\") " Oct 30 22:09:12.413052 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:12.413024 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b5c2b3-a188-48ea-9352-9cb449f19335-maas-api-openshift-ca" pod "d6b5c2b3-a188-48ea-9352-9cb449f19335" (UID: "d6b5c2b3-a188-48ea-9352-9cb449f19335"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:12.415429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:12.415401 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b5c2b3-a188-48ea-9352-9cb449f19335-kube-api-access-sqqrk" pod "d6b5c2b3-a188-48ea-9352-9cb449f19335" (UID: "d6b5c2b3-a188-48ea-9352-9cb449f19335"). InnerVolumeSpecName "kube-api-access-sqqrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:12.426800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:12.426773 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b5c2b3-a188-48ea-9352-9cb449f19335-maas-api-xks-ca" pod "d6b5c2b3-a188-48ea-9352-9cb449f19335" (UID: "d6b5c2b3-a188-48ea-9352-9cb449f19335"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:12.514118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:12.514087 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqqrk\" (UniqueName: \"kubernetes.io/projected/d6b5c2b3-a188-48ea-9352-9cb449f19335-kube-api-access-sqqrk\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:12.514118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:12.514114 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b5c2b3-a188-48ea-9352-9cb449f19335-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:12.514118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:12.514125 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b5c2b3-a188-48ea-9352-9cb449f19335-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:13.061833 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:13.061775 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6"] Oct 30 22:09:13.101883 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:13.101858 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94d3ee89-p5jg6"] Oct 30 22:09:13.119287 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:13.119267 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj" Oct 30 22:09:13.119719 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:13.119277 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj" event={"ID":"d6b5c2b3-a188-48ea-9352-9cb449f19335","Type":"ContainerDied","Data":"3b7164653fd54f3e012599b354782062ffc317036a594d862fd830f7469a3a28"} Oct 30 22:09:13.119719 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:13.119419 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b7164653fd54f3e012599b354782062ffc317036a594d862fd830f7469a3a28" Oct 30 22:09:13.121379 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:13.121356 4335 generic.go:356] "Generic (PLEG): container finished" podID="8797e804-75ad-40d8-87c8-dcb6152e9af8" containerID="0a9d187d683d679e6aff08a10dfbbe907325d061481068887e3b95ef6c1cb0ad" exitCode=0 Oct 30 22:09:13.121494 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:13.121389 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws" event={"ID":"8797e804-75ad-40d8-87c8-dcb6152e9af8","Type":"ContainerDied","Data":"0a9d187d683d679e6aff08a10dfbbe907325d061481068887e3b95ef6c1cb0ad"} Oct 30 22:09:14.053575 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.053542 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv"] Oct 30 22:09:14.102421 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.102392 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a54b8c68-7xlbv"] Oct 30 22:09:14.238828 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.238791 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc"] Oct 30 22:09:14.239901 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.239878 4335 cpu_manager.go:414] "containerMap: removing container" podUID="d6b5c2b3-a188-48ea-9352-9cb449f19335" containerName="revoke-keys" Oct 30 22:09:14.240032 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.239903 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b5c2b3-a188-48ea-9352-9cb449f19335" containerName="revoke-keys" Oct 30 22:09:14.240119 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.240103 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b5c2b3-a188-48ea-9352-9cb449f19335" containerName="revoke-keys" Oct 30 22:09:14.245070 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.245049 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc" Oct 30 22:09:14.297063 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.297042 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws" Oct 30 22:09:14.298253 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.298230 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc"] Oct 30 22:09:14.335840 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.335778 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8797e804-75ad-40d8-87c8-dcb6152e9af8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8797e804-75ad-40d8-87c8-dcb6152e9af8-maas-api-openshift-ca\") pod \"8797e804-75ad-40d8-87c8-dcb6152e9af8\" (UID: \"8797e804-75ad-40d8-87c8-dcb6152e9af8\") " Oct 30 22:09:14.335949 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.335904 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8797e804-75ad-40d8-87c8-dcb6152e9af8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8797e804-75ad-40d8-87c8-dcb6152e9af8-maas-api-xks-ca\") pod \"8797e804-75ad-40d8-87c8-dcb6152e9af8\" (UID: \"8797e804-75ad-40d8-87c8-dcb6152e9af8\") " Oct 30 22:09:14.336012 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.335978 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8797e804-75ad-40d8-87c8-dcb6152e9af8-kube-api-access-cp8nw\" (UniqueName: \"kubernetes.io/projected/8797e804-75ad-40d8-87c8-dcb6152e9af8-kube-api-access-cp8nw\") pod \"8797e804-75ad-40d8-87c8-dcb6152e9af8\" (UID: \"8797e804-75ad-40d8-87c8-dcb6152e9af8\") " Oct 30 22:09:14.336194 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.336166 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8797e804-75ad-40d8-87c8-dcb6152e9af8-maas-api-openshift-ca" pod "8797e804-75ad-40d8-87c8-dcb6152e9af8" (UID: "8797e804-75ad-40d8-87c8-dcb6152e9af8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:14.336285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.336263 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/952bbcca-025c-40de-be76-21b90316c51a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86116beb-9vvzc\" (UID: \"952bbcca-025c-40de-be76-21b90316c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc" Oct 30 22:09:14.336372 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.336354 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9h25\" (UniqueName: \"kubernetes.io/projected/952bbcca-025c-40de-be76-21b90316c51a-kube-api-access-j9h25\") pod \"maas-api-revoke-sub-86116beb-9vvzc\" (UID: \"952bbcca-025c-40de-be76-21b90316c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc" Oct 30 22:09:14.336770 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.336632 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/952bbcca-025c-40de-be76-21b90316c51a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86116beb-9vvzc\" (UID: \"952bbcca-025c-40de-be76-21b90316c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc" Oct 30 22:09:14.336897 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.336880 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8797e804-75ad-40d8-87c8-dcb6152e9af8-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:14.338238 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.338219 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8797e804-75ad-40d8-87c8-dcb6152e9af8-kube-api-access-cp8nw" pod "8797e804-75ad-40d8-87c8-dcb6152e9af8" (UID: "8797e804-75ad-40d8-87c8-dcb6152e9af8"). InnerVolumeSpecName "kube-api-access-cp8nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:14.338600 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.338585 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8797e804-75ad-40d8-87c8-dcb6152e9af8-maas-api-xks-ca" pod "8797e804-75ad-40d8-87c8-dcb6152e9af8" (UID: "8797e804-75ad-40d8-87c8-dcb6152e9af8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:14.437963 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.437932 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9h25\" (UniqueName: \"kubernetes.io/projected/952bbcca-025c-40de-be76-21b90316c51a-kube-api-access-j9h25\") pod \"maas-api-revoke-sub-86116beb-9vvzc\" (UID: \"952bbcca-025c-40de-be76-21b90316c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc" Oct 30 22:09:14.438119 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.438100 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/952bbcca-025c-40de-be76-21b90316c51a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86116beb-9vvzc\" (UID: \"952bbcca-025c-40de-be76-21b90316c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc" Oct 30 22:09:14.438234 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.438219 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/952bbcca-025c-40de-be76-21b90316c51a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86116beb-9vvzc\" (UID: \"952bbcca-025c-40de-be76-21b90316c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc" Oct 30 22:09:14.438318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.438303 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8797e804-75ad-40d8-87c8-dcb6152e9af8-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:14.438367 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.438325 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp8nw\" (UniqueName: \"kubernetes.io/projected/8797e804-75ad-40d8-87c8-dcb6152e9af8-kube-api-access-cp8nw\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:14.438908 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.438850 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/952bbcca-025c-40de-be76-21b90316c51a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86116beb-9vvzc\" (UID: \"952bbcca-025c-40de-be76-21b90316c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc" Oct 30 22:09:14.440449 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.440423 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/952bbcca-025c-40de-be76-21b90316c51a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86116beb-9vvzc\" (UID: \"952bbcca-025c-40de-be76-21b90316c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc" Oct 30 22:09:14.504238 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.504211 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9h25\" (UniqueName: \"kubernetes.io/projected/952bbcca-025c-40de-be76-21b90316c51a-kube-api-access-j9h25\") pod \"maas-api-revoke-sub-86116beb-9vvzc\" (UID: \"952bbcca-025c-40de-be76-21b90316c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc" Oct 30 22:09:14.561655 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.561630 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc" Oct 30 22:09:14.943961 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:09:14.943923 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952bbcca_025c_40de_be76_21b90316c51a.slice/crio-1a2c1e1b4daf33c478f055fa488312f70415055a18c1b82db7c330d795e169a4 WatchSource:0}: Error finding container 1a2c1e1b4daf33c478f055fa488312f70415055a18c1b82db7c330d795e169a4: Status 404 returned error can't find the container with id 1a2c1e1b4daf33c478f055fa488312f70415055a18c1b82db7c330d795e169a4 Oct 30 22:09:14.944689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:14.944667 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc"] Oct 30 22:09:15.059148 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:15.059124 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ab13cda-bcde-45d7-bb98-ee11ddcdfc49" path="/var/lib/kubelet/pods/9ab13cda-bcde-45d7-bb98-ee11ddcdfc49/volumes" Oct 30 22:09:15.059554 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:15.059540 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf71281e-6d19-47db-9e68-cd63215f0824" path="/var/lib/kubelet/pods/cf71281e-6d19-47db-9e68-cd63215f0824/volumes" Oct 30 22:09:15.140349 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:15.140328 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws" Oct 30 22:09:15.140349 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:15.140334 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws" event={"ID":"8797e804-75ad-40d8-87c8-dcb6152e9af8","Type":"ContainerDied","Data":"0cee531c8e5732f231ee25a4d75f4f3ce77a279c26a6ebd829ead72348ae2523"} Oct 30 22:09:15.140500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:15.140382 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cee531c8e5732f231ee25a4d75f4f3ce77a279c26a6ebd829ead72348ae2523" Oct 30 22:09:15.141881 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:15.141861 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc" event={"ID":"952bbcca-025c-40de-be76-21b90316c51a","Type":"ContainerStarted","Data":"1a2c1e1b4daf33c478f055fa488312f70415055a18c1b82db7c330d795e169a4"} Oct 30 22:09:16.154205 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:16.154170 4335 generic.go:356] "Generic (PLEG): container finished" podID="952bbcca-025c-40de-be76-21b90316c51a" containerID="309c348efdc09ccef6a68b1821b64c94f1adf57713fe72fd70d41177608b68e4" exitCode=0 Oct 30 22:09:16.154695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:16.154252 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc" event={"ID":"952bbcca-025c-40de-be76-21b90316c51a","Type":"ContainerDied","Data":"309c348efdc09ccef6a68b1821b64c94f1adf57713fe72fd70d41177608b68e4"} Oct 30 22:09:17.334374 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:17.334348 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc" Oct 30 22:09:17.381136 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:17.381109 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c"] Oct 30 22:09:17.381313 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:17.381294 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" podUID="8ec3aebd-024a-420b-a207-9ca1d8982d29" containerName="istio-proxy" containerID="cri-o://1932847dd06d0a4fb3029dd27f518af3d99aab9c4630648af2b7b33afad16edd" gracePeriod=30 Oct 30 22:09:17.481262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:17.481182 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952bbcca-025c-40de-be76-21b90316c51a-kube-api-access-j9h25\" (UniqueName: \"kubernetes.io/projected/952bbcca-025c-40de-be76-21b90316c51a-kube-api-access-j9h25\") pod \"952bbcca-025c-40de-be76-21b90316c51a\" (UID: \"952bbcca-025c-40de-be76-21b90316c51a\") " Oct 30 22:09:17.481812 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:17.481361 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/952bbcca-025c-40de-be76-21b90316c51a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/952bbcca-025c-40de-be76-21b90316c51a-maas-api-openshift-ca\") pod \"952bbcca-025c-40de-be76-21b90316c51a\" (UID: \"952bbcca-025c-40de-be76-21b90316c51a\") " Oct 30 22:09:17.482045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:17.482019 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/952bbcca-025c-40de-be76-21b90316c51a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/952bbcca-025c-40de-be76-21b90316c51a-maas-api-xks-ca\") pod \"952bbcca-025c-40de-be76-21b90316c51a\" (UID: \"952bbcca-025c-40de-be76-21b90316c51a\") " Oct 30 22:09:17.482045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:17.481777 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/952bbcca-025c-40de-be76-21b90316c51a-maas-api-openshift-ca" pod "952bbcca-025c-40de-be76-21b90316c51a" (UID: "952bbcca-025c-40de-be76-21b90316c51a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:17.482873 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:17.482847 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/952bbcca-025c-40de-be76-21b90316c51a-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:17.484654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:17.484383 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/952bbcca-025c-40de-be76-21b90316c51a-maas-api-xks-ca" pod "952bbcca-025c-40de-be76-21b90316c51a" (UID: "952bbcca-025c-40de-be76-21b90316c51a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:17.484980 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:17.484952 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952bbcca-025c-40de-be76-21b90316c51a-kube-api-access-j9h25" pod "952bbcca-025c-40de-be76-21b90316c51a" (UID: "952bbcca-025c-40de-be76-21b90316c51a"). InnerVolumeSpecName "kube-api-access-j9h25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:17.584267 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:17.584235 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9h25\" (UniqueName: \"kubernetes.io/projected/952bbcca-025c-40de-be76-21b90316c51a-kube-api-access-j9h25\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:17.584267 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:17.584266 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/952bbcca-025c-40de-be76-21b90316c51a-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:18.174942 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:18.174917 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc" Oct 30 22:09:18.175107 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:18.174951 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc" event={"ID":"952bbcca-025c-40de-be76-21b90316c51a","Type":"ContainerDied","Data":"1a2c1e1b4daf33c478f055fa488312f70415055a18c1b82db7c330d795e169a4"} Oct 30 22:09:18.175107 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:18.174996 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a2c1e1b4daf33c478f055fa488312f70415055a18c1b82db7c330d795e169a4" Oct 30 22:09:19.438725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.438682 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2"] Oct 30 22:09:19.439648 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.439631 4335 cpu_manager.go:414] "containerMap: removing container" podUID="8797e804-75ad-40d8-87c8-dcb6152e9af8" containerName="revoke-keys" Oct 30 22:09:19.439648 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.439647 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8797e804-75ad-40d8-87c8-dcb6152e9af8" containerName="revoke-keys" Oct 30 22:09:19.439736 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.439692 4335 cpu_manager.go:414] "containerMap: removing container" podUID="952bbcca-025c-40de-be76-21b90316c51a" containerName="revoke-keys" Oct 30 22:09:19.439736 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.439698 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952bbcca-025c-40de-be76-21b90316c51a" containerName="revoke-keys" Oct 30 22:09:19.439811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.439801 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="952bbcca-025c-40de-be76-21b90316c51a" containerName="revoke-keys" Oct 30 22:09:19.439842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.439814 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="8797e804-75ad-40d8-87c8-dcb6152e9af8" containerName="revoke-keys" Oct 30 22:09:19.443624 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.443598 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2" Oct 30 22:09:19.497643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.497603 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2"] Oct 30 22:09:19.540597 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.540572 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:19.540828 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.540791 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:09:19.540828 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.540680 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:19.540828 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.540599 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:19.611034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.610958 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8vfh\" (UniqueName: \"kubernetes.io/projected/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-kube-api-access-v8vfh\") pod \"maas-api-revoke-sub-fed3da06-wh9q2\" (UID: \"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2" Oct 30 22:09:19.611325 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.611306 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fed3da06-wh9q2\" (UID: \"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2" Oct 30 22:09:19.611408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.611395 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fed3da06-wh9q2\" (UID: \"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2" Oct 30 22:09:19.712645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.712549 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fed3da06-wh9q2\" (UID: \"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2" Oct 30 22:09:19.712828 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.712678 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8vfh\" (UniqueName: \"kubernetes.io/projected/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-kube-api-access-v8vfh\") pod \"maas-api-revoke-sub-fed3da06-wh9q2\" (UID: \"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2" Oct 30 22:09:19.712941 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.712923 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fed3da06-wh9q2\" (UID: \"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2" Oct 30 22:09:19.713703 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.713671 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fed3da06-wh9q2\" (UID: \"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2" Oct 30 22:09:19.715156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.715134 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fed3da06-wh9q2\" (UID: \"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2" Oct 30 22:09:19.774165 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:19.774136 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8vfh\" (UniqueName: \"kubernetes.io/projected/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-kube-api-access-v8vfh\") pod \"maas-api-revoke-sub-fed3da06-wh9q2\" (UID: \"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2" Oct 30 22:09:20.057705 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:20.057681 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2" Oct 30 22:09:20.448111 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:09:20.448070 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6b1c09_4a5d_403c_a6e9_e4b52bf503bb.slice/crio-99b4abaebd94eb134f387d3ceb8649f08ada6808fbae8498b52dd84ba203cc2b WatchSource:0}: Error finding container 99b4abaebd94eb134f387d3ceb8649f08ada6808fbae8498b52dd84ba203cc2b: Status 404 returned error can't find the container with id 99b4abaebd94eb134f387d3ceb8649f08ada6808fbae8498b52dd84ba203cc2b Oct 30 22:09:20.452244 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:20.452221 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2"] Oct 30 22:09:20.978692 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:20.978624 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-gvj5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:09:20.978692 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:20.978672 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" podUID="8ec3aebd-024a-420b-a207-9ca1d8982d29" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:09:21.208279 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:21.208245 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2" event={"ID":"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb","Type":"ContainerStarted","Data":"5b2a9cdd931d3c3332cad1c50b60a62a0a2037b3de9cf103ee2bb06aa3ea59ba"} Oct 30 22:09:21.208468 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:21.208311 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2" event={"ID":"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb","Type":"ContainerStarted","Data":"99b4abaebd94eb134f387d3ceb8649f08ada6808fbae8498b52dd84ba203cc2b"} Oct 30 22:09:21.326269 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:21.326221 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2" podStartSLOduration=2.326207507 podStartE2EDuration="2.326207507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:21.323446416 +0000 UTC m=+1494.423878157" watchObservedRunningTime="2026-10-30 22:09:21.326207507 +0000 UTC m=+1494.426639239" Oct 30 22:09:22.054931 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.054896 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247"] Oct 30 22:09:22.104517 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.104480 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89834086-m5247"] Oct 30 22:09:22.222602 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.222567 4335 generic.go:356] "Generic (PLEG): container finished" podID="1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb" containerID="5b2a9cdd931d3c3332cad1c50b60a62a0a2037b3de9cf103ee2bb06aa3ea59ba" exitCode=0 Oct 30 22:09:22.222791 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.222670 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2" event={"ID":"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb","Type":"ContainerDied","Data":"5b2a9cdd931d3c3332cad1c50b60a62a0a2037b3de9cf103ee2bb06aa3ea59ba"} Oct 30 22:09:22.389255 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.389171 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm"] Oct 30 22:09:22.413645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.410029 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm" Oct 30 22:09:22.450078 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.450033 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm"] Oct 30 22:09:22.554057 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.554020 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aada93e-1b88-4bae-993e-fdc80c3227b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b632cbaa-l9ndm\" (UID: \"5aada93e-1b88-4bae-993e-fdc80c3227b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm" Oct 30 22:09:22.554238 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.554121 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aada93e-1b88-4bae-993e-fdc80c3227b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b632cbaa-l9ndm\" (UID: \"5aada93e-1b88-4bae-993e-fdc80c3227b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm" Oct 30 22:09:22.554238 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.554228 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmqt5\" (UniqueName: \"kubernetes.io/projected/5aada93e-1b88-4bae-993e-fdc80c3227b9-kube-api-access-bmqt5\") pod \"maas-api-revoke-sub-b632cbaa-l9ndm\" (UID: \"5aada93e-1b88-4bae-993e-fdc80c3227b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm" Oct 30 22:09:22.656123 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.656074 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aada93e-1b88-4bae-993e-fdc80c3227b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b632cbaa-l9ndm\" (UID: \"5aada93e-1b88-4bae-993e-fdc80c3227b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm" Oct 30 22:09:22.656246 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.656149 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aada93e-1b88-4bae-993e-fdc80c3227b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b632cbaa-l9ndm\" (UID: \"5aada93e-1b88-4bae-993e-fdc80c3227b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm" Oct 30 22:09:22.656327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.656312 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmqt5\" (UniqueName: \"kubernetes.io/projected/5aada93e-1b88-4bae-993e-fdc80c3227b9-kube-api-access-bmqt5\") pod \"maas-api-revoke-sub-b632cbaa-l9ndm\" (UID: \"5aada93e-1b88-4bae-993e-fdc80c3227b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm" Oct 30 22:09:22.656771 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.656735 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aada93e-1b88-4bae-993e-fdc80c3227b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b632cbaa-l9ndm\" (UID: \"5aada93e-1b88-4bae-993e-fdc80c3227b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm" Oct 30 22:09:22.658703 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.658669 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aada93e-1b88-4bae-993e-fdc80c3227b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b632cbaa-l9ndm\" (UID: \"5aada93e-1b88-4bae-993e-fdc80c3227b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm" Oct 30 22:09:22.690065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.690043 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 22:09:22.717739 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.717714 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmqt5\" (UniqueName: \"kubernetes.io/projected/5aada93e-1b88-4bae-993e-fdc80c3227b9-kube-api-access-bmqt5\") pod \"maas-api-revoke-sub-b632cbaa-l9ndm\" (UID: \"5aada93e-1b88-4bae-993e-fdc80c3227b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm" Oct 30 22:09:22.796268 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.796244 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm" Oct 30 22:09:22.859068 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.858893 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-podinfo\") pod \"8ec3aebd-024a-420b-a207-9ca1d8982d29\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " Oct 30 22:09:22.859068 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.858994 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-data\") pod \"8ec3aebd-024a-420b-a207-9ca1d8982d29\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " Oct 30 22:09:22.861504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.859524 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-token\" (UniqueName: \"kubernetes.io/projected/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-token\") pod \"8ec3aebd-024a-420b-a207-9ca1d8982d29\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " Oct 30 22:09:22.861504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.859725 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-workload-socket\") pod \"8ec3aebd-024a-420b-a207-9ca1d8982d29\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " Oct 30 22:09:22.861504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.859895 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-envoy\") pod \"8ec3aebd-024a-420b-a207-9ca1d8982d29\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " Oct 30 22:09:22.861504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.859969 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-credential-socket\") pod \"8ec3aebd-024a-420b-a207-9ca1d8982d29\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " Oct 30 22:09:22.861504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.860012 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ec3aebd-024a-420b-a207-9ca1d8982d29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec3aebd-024a-420b-a207-9ca1d8982d29-istiod-ca-cert\") pod \"8ec3aebd-024a-420b-a207-9ca1d8982d29\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " Oct 30 22:09:22.861504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.860041 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec3aebd-024a-420b-a207-9ca1d8982d29-kube-api-access-ft8xr\" (UniqueName: \"kubernetes.io/projected/8ec3aebd-024a-420b-a207-9ca1d8982d29-kube-api-access-ft8xr\") pod \"8ec3aebd-024a-420b-a207-9ca1d8982d29\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " Oct 30 22:09:22.861504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.860069 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-workload-certs\") pod \"8ec3aebd-024a-420b-a207-9ca1d8982d29\" (UID: \"8ec3aebd-024a-420b-a207-9ca1d8982d29\") " Oct 30 22:09:22.861504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.861015 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-workload-certs" pod "8ec3aebd-024a-420b-a207-9ca1d8982d29" (UID: "8ec3aebd-024a-420b-a207-9ca1d8982d29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:22.861504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.861186 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-data" pod "8ec3aebd-024a-420b-a207-9ca1d8982d29" (UID: "8ec3aebd-024a-420b-a207-9ca1d8982d29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:22.864130 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.861888 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-credential-socket" pod "8ec3aebd-024a-420b-a207-9ca1d8982d29" (UID: "8ec3aebd-024a-420b-a207-9ca1d8982d29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:22.864130 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.862082 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-workload-socket" pod "8ec3aebd-024a-420b-a207-9ca1d8982d29" (UID: "8ec3aebd-024a-420b-a207-9ca1d8982d29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:22.864836 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.864799 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-podinfo" pod "8ec3aebd-024a-420b-a207-9ca1d8982d29" (UID: "8ec3aebd-024a-420b-a207-9ca1d8982d29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:09:22.865392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.865365 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec3aebd-024a-420b-a207-9ca1d8982d29-istiod-ca-cert" pod "8ec3aebd-024a-420b-a207-9ca1d8982d29" (UID: "8ec3aebd-024a-420b-a207-9ca1d8982d29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:22.867990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.867962 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec3aebd-024a-420b-a207-9ca1d8982d29-kube-api-access-ft8xr" pod "8ec3aebd-024a-420b-a207-9ca1d8982d29" (UID: "8ec3aebd-024a-420b-a207-9ca1d8982d29"). InnerVolumeSpecName "kube-api-access-ft8xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:22.868126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.867986 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-envoy" pod "8ec3aebd-024a-420b-a207-9ca1d8982d29" (UID: "8ec3aebd-024a-420b-a207-9ca1d8982d29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:22.868126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.868008 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-token" pod "8ec3aebd-024a-420b-a207-9ca1d8982d29" (UID: "8ec3aebd-024a-420b-a207-9ca1d8982d29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:22.961747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.961670 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:22.961747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.961697 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:22.961747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.961708 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:22.961747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.961717 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:22.961747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.961726 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:22.961747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.961735 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec3aebd-024a-420b-a207-9ca1d8982d29-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:22.961747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.961744 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft8xr\" (UniqueName: \"kubernetes.io/projected/8ec3aebd-024a-420b-a207-9ca1d8982d29-kube-api-access-ft8xr\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:22.961747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.961754 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec3aebd-024a-420b-a207-9ca1d8982d29-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:22.962271 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:22.961762 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec3aebd-024a-420b-a207-9ca1d8982d29-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:23.059303 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.059275 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9" path="/var/lib/kubelet/pods/9e3145bc-4a50-4b9e-bfa1-e81ed0cac3f9/volumes" Oct 30 22:09:23.203270 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:09:23.203240 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aada93e_1b88_4bae_993e_fdc80c3227b9.slice/crio-edce5f77d4a284e91d3c299387d5cccf42aaa2e7567e3d34c33b7c13f6a8b2ea WatchSource:0}: Error finding container edce5f77d4a284e91d3c299387d5cccf42aaa2e7567e3d34c33b7c13f6a8b2ea: Status 404 returned error can't find the container with id edce5f77d4a284e91d3c299387d5cccf42aaa2e7567e3d34c33b7c13f6a8b2ea Oct 30 22:09:23.205713 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.205688 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm"] Oct 30 22:09:23.233867 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.233844 4335 generic.go:356] "Generic (PLEG): container finished" podID="8ec3aebd-024a-420b-a207-9ca1d8982d29" containerID="1932847dd06d0a4fb3029dd27f518af3d99aab9c4630648af2b7b33afad16edd" exitCode=0 Oct 30 22:09:23.233987 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.233902 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" Oct 30 22:09:23.233987 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.233929 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" event={"ID":"8ec3aebd-024a-420b-a207-9ca1d8982d29","Type":"ContainerDied","Data":"1932847dd06d0a4fb3029dd27f518af3d99aab9c4630648af2b7b33afad16edd"} Oct 30 22:09:23.234102 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.233996 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c" event={"ID":"8ec3aebd-024a-420b-a207-9ca1d8982d29","Type":"ContainerDied","Data":"24b9d06d6e42de4b91da04fae617c38cefeb58f6b343e3ea4998d63370d37043"} Oct 30 22:09:23.234102 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.234016 4335 scope.go:122] "RemoveContainer" containerID="1932847dd06d0a4fb3029dd27f518af3d99aab9c4630648af2b7b33afad16edd" Oct 30 22:09:23.235813 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.235760 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm" event={"ID":"5aada93e-1b88-4bae-993e-fdc80c3227b9","Type":"ContainerStarted","Data":"edce5f77d4a284e91d3c299387d5cccf42aaa2e7567e3d34c33b7c13f6a8b2ea"} Oct 30 22:09:23.248537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.248504 4335 scope.go:122] "RemoveContainer" containerID="1932847dd06d0a4fb3029dd27f518af3d99aab9c4630648af2b7b33afad16edd" Oct 30 22:09:23.248845 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:09:23.248824 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1932847dd06d0a4fb3029dd27f518af3d99aab9c4630648af2b7b33afad16edd\": container with ID starting with 1932847dd06d0a4fb3029dd27f518af3d99aab9c4630648af2b7b33afad16edd not found: ID does not exist" containerID="1932847dd06d0a4fb3029dd27f518af3d99aab9c4630648af2b7b33afad16edd" Oct 30 22:09:23.248916 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.248849 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1932847dd06d0a4fb3029dd27f518af3d99aab9c4630648af2b7b33afad16edd"} err="failed to get container status \"1932847dd06d0a4fb3029dd27f518af3d99aab9c4630648af2b7b33afad16edd\": rpc error: code = NotFound desc = could not find container \"1932847dd06d0a4fb3029dd27f518af3d99aab9c4630648af2b7b33afad16edd\": container with ID starting with 1932847dd06d0a4fb3029dd27f518af3d99aab9c4630648af2b7b33afad16edd not found: ID does not exist" Oct 30 22:09:23.394880 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.394852 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2" Oct 30 22:09:23.406317 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.406288 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c"] Oct 30 22:09:23.415464 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.415434 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gvj5c"] Oct 30 22:09:23.570710 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.570685 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-kube-api-access-v8vfh\" (UniqueName: \"kubernetes.io/projected/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-kube-api-access-v8vfh\") pod \"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb\" (UID: \"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb\") " Oct 30 22:09:23.570860 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.570790 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-maas-api-xks-ca\") pod \"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb\" (UID: \"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb\") " Oct 30 22:09:23.570906 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.570868 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-maas-api-openshift-ca\") pod \"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb\" (UID: \"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb\") " Oct 30 22:09:23.571224 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.571197 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-maas-api-openshift-ca" pod "1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb" (UID: "1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:23.571373 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.571356 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:23.572866 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.572846 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-kube-api-access-v8vfh" pod "1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb" (UID: "1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb"). InnerVolumeSpecName "kube-api-access-v8vfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:23.572974 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.572903 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-maas-api-xks-ca" pod "1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb" (UID: "1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:23.673285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.673261 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8vfh\" (UniqueName: \"kubernetes.io/projected/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-kube-api-access-v8vfh\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:23.673285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:23.673284 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:24.251512 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.251472 4335 generic.go:356] "Generic (PLEG): container finished" podID="5aada93e-1b88-4bae-993e-fdc80c3227b9" containerID="60f57460c318d5f41dd8f26d288eb80dcf9ae1d040f009ff220b040c2a9afa57" exitCode=0 Oct 30 22:09:24.251937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.251525 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm" event={"ID":"5aada93e-1b88-4bae-993e-fdc80c3227b9","Type":"ContainerDied","Data":"60f57460c318d5f41dd8f26d288eb80dcf9ae1d040f009ff220b040c2a9afa57"} Oct 30 22:09:24.253254 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.253233 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2" Oct 30 22:09:24.253369 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.253269 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2" event={"ID":"1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb","Type":"ContainerDied","Data":"99b4abaebd94eb134f387d3ceb8649f08ada6808fbae8498b52dd84ba203cc2b"} Oct 30 22:09:24.253369 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.253317 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99b4abaebd94eb134f387d3ceb8649f08ada6808fbae8498b52dd84ba203cc2b" Oct 30 22:09:24.862557 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.862524 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6"] Oct 30 22:09:24.863227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.863210 4335 cpu_manager.go:414] "containerMap: removing container" podUID="8ec3aebd-024a-420b-a207-9ca1d8982d29" containerName="istio-proxy" Oct 30 22:09:24.863227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.863228 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec3aebd-024a-420b-a207-9ca1d8982d29" containerName="istio-proxy" Oct 30 22:09:24.863325 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.863256 4335 cpu_manager.go:414] "containerMap: removing container" podUID="1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb" containerName="revoke-keys" Oct 30 22:09:24.863325 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.863261 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb" containerName="revoke-keys" Oct 30 22:09:24.863406 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.863396 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec3aebd-024a-420b-a207-9ca1d8982d29" containerName="istio-proxy" Oct 30 22:09:24.863456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.863411 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb" containerName="revoke-keys" Oct 30 22:09:24.867120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.867104 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6" Oct 30 22:09:24.892084 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.892058 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de62402c-594f-4a89-910c-434540713997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-934ee986-tmjx6\" (UID: \"de62402c-594f-4a89-910c-434540713997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6" Oct 30 22:09:24.892220 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.892171 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl78m\" (UniqueName: \"kubernetes.io/projected/de62402c-594f-4a89-910c-434540713997-kube-api-access-dl78m\") pod \"maas-api-revoke-sub-934ee986-tmjx6\" (UID: \"de62402c-594f-4a89-910c-434540713997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6" Oct 30 22:09:24.892290 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.892254 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de62402c-594f-4a89-910c-434540713997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-934ee986-tmjx6\" (UID: \"de62402c-594f-4a89-910c-434540713997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6" Oct 30 22:09:24.926399 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.926364 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6"] Oct 30 22:09:24.993390 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.993361 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de62402c-594f-4a89-910c-434540713997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-934ee986-tmjx6\" (UID: \"de62402c-594f-4a89-910c-434540713997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6" Oct 30 22:09:24.993518 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.993459 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl78m\" (UniqueName: \"kubernetes.io/projected/de62402c-594f-4a89-910c-434540713997-kube-api-access-dl78m\") pod \"maas-api-revoke-sub-934ee986-tmjx6\" (UID: \"de62402c-594f-4a89-910c-434540713997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6" Oct 30 22:09:24.993566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.993543 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de62402c-594f-4a89-910c-434540713997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-934ee986-tmjx6\" (UID: \"de62402c-594f-4a89-910c-434540713997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6" Oct 30 22:09:24.994195 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.994171 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de62402c-594f-4a89-910c-434540713997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-934ee986-tmjx6\" (UID: \"de62402c-594f-4a89-910c-434540713997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6" Oct 30 22:09:24.995968 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:24.995946 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de62402c-594f-4a89-910c-434540713997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-934ee986-tmjx6\" (UID: \"de62402c-594f-4a89-910c-434540713997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6" Oct 30 22:09:25.054431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:25.054407 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl78m\" (UniqueName: \"kubernetes.io/projected/de62402c-594f-4a89-910c-434540713997-kube-api-access-dl78m\") pod \"maas-api-revoke-sub-934ee986-tmjx6\" (UID: \"de62402c-594f-4a89-910c-434540713997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6" Oct 30 22:09:25.061901 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:25.061878 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ec3aebd-024a-420b-a207-9ca1d8982d29" path="/var/lib/kubelet/pods/8ec3aebd-024a-420b-a207-9ca1d8982d29/volumes" Oct 30 22:09:25.181365 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:25.181294 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6" Oct 30 22:09:25.352932 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:09:25.352674 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:09:25.466065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:25.466037 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm" Oct 30 22:09:25.501059 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:25.501027 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aada93e-1b88-4bae-993e-fdc80c3227b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aada93e-1b88-4bae-993e-fdc80c3227b9-maas-api-xks-ca\") pod \"5aada93e-1b88-4bae-993e-fdc80c3227b9\" (UID: \"5aada93e-1b88-4bae-993e-fdc80c3227b9\") " Oct 30 22:09:25.501176 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:25.501160 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aada93e-1b88-4bae-993e-fdc80c3227b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aada93e-1b88-4bae-993e-fdc80c3227b9-maas-api-openshift-ca\") pod \"5aada93e-1b88-4bae-993e-fdc80c3227b9\" (UID: \"5aada93e-1b88-4bae-993e-fdc80c3227b9\") " Oct 30 22:09:25.501260 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:25.501242 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aada93e-1b88-4bae-993e-fdc80c3227b9-kube-api-access-bmqt5\" (UniqueName: \"kubernetes.io/projected/5aada93e-1b88-4bae-993e-fdc80c3227b9-kube-api-access-bmqt5\") pod \"5aada93e-1b88-4bae-993e-fdc80c3227b9\" (UID: \"5aada93e-1b88-4bae-993e-fdc80c3227b9\") " Oct 30 22:09:25.501547 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:25.501520 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aada93e-1b88-4bae-993e-fdc80c3227b9-maas-api-openshift-ca" pod "5aada93e-1b88-4bae-993e-fdc80c3227b9" (UID: "5aada93e-1b88-4bae-993e-fdc80c3227b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:25.501918 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:25.501883 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aada93e-1b88-4bae-993e-fdc80c3227b9-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:25.503208 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:25.503172 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aada93e-1b88-4bae-993e-fdc80c3227b9-maas-api-xks-ca" pod "5aada93e-1b88-4bae-993e-fdc80c3227b9" (UID: "5aada93e-1b88-4bae-993e-fdc80c3227b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:25.503514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:25.503487 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aada93e-1b88-4bae-993e-fdc80c3227b9-kube-api-access-bmqt5" pod "5aada93e-1b88-4bae-993e-fdc80c3227b9" (UID: "5aada93e-1b88-4bae-993e-fdc80c3227b9"). InnerVolumeSpecName "kube-api-access-bmqt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:25.571814 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:09:25.571785 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde62402c_594f_4a89_910c_434540713997.slice/crio-82a8d2e54ccdd272ede14a2e27d8c520f1d527917e788d1dbfa34f9ebdc3d3ac WatchSource:0}: Error finding container 82a8d2e54ccdd272ede14a2e27d8c520f1d527917e788d1dbfa34f9ebdc3d3ac: Status 404 returned error can't find the container with id 82a8d2e54ccdd272ede14a2e27d8c520f1d527917e788d1dbfa34f9ebdc3d3ac Oct 30 22:09:25.576789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:25.576766 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6"] Oct 30 22:09:25.603780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:25.603758 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmqt5\" (UniqueName: \"kubernetes.io/projected/5aada93e-1b88-4bae-993e-fdc80c3227b9-kube-api-access-bmqt5\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:25.603860 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:25.603786 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aada93e-1b88-4bae-993e-fdc80c3227b9-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:26.052951 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.052924 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq"] Oct 30 22:09:26.058052 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.057994 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl"] Oct 30 22:09:26.062383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.062360 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f01cf7-rfbcq"] Oct 30 22:09:26.067783 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.067764 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f43fbfe-zbkzl"] Oct 30 22:09:26.279953 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.279916 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6" event={"ID":"de62402c-594f-4a89-910c-434540713997","Type":"ContainerStarted","Data":"15c10858297f400351b61f5f9421652a9225c83c27d7371c0ac4ff6fb5c708f8"} Oct 30 22:09:26.280133 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.279973 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6" event={"ID":"de62402c-594f-4a89-910c-434540713997","Type":"ContainerStarted","Data":"82a8d2e54ccdd272ede14a2e27d8c520f1d527917e788d1dbfa34f9ebdc3d3ac"} Oct 30 22:09:26.281608 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.281581 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm" event={"ID":"5aada93e-1b88-4bae-993e-fdc80c3227b9","Type":"ContainerDied","Data":"edce5f77d4a284e91d3c299387d5cccf42aaa2e7567e3d34c33b7c13f6a8b2ea"} Oct 30 22:09:26.281745 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.281639 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm" Oct 30 22:09:26.281745 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.281652 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edce5f77d4a284e91d3c299387d5cccf42aaa2e7567e3d34c33b7c13f6a8b2ea" Oct 30 22:09:26.301211 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.301180 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25"] Oct 30 22:09:26.301921 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.301903 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5aada93e-1b88-4bae-993e-fdc80c3227b9" containerName="revoke-keys" Oct 30 22:09:26.301921 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.301920 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aada93e-1b88-4bae-993e-fdc80c3227b9" containerName="revoke-keys" Oct 30 22:09:26.302081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.302026 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aada93e-1b88-4bae-993e-fdc80c3227b9" containerName="revoke-keys" Oct 30 22:09:26.326987 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.326918 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.331632 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.330322 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx"] Oct 30 22:09:26.334854 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.334827 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.367454 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.367424 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25"] Oct 30 22:09:26.411373 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.398662 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6" podStartSLOduration=2.398650439 podStartE2EDuration="2.398650439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:26.395349703 +0000 UTC m=+1499.495781435" watchObservedRunningTime="2026-10-30 22:09:26.398650439 +0000 UTC m=+1499.499082170" Oct 30 22:09:26.411533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.411409 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx"] Oct 30 22:09:26.524217 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.524184 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-data\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.524217 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.524223 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-envoy\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.524461 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.524244 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-token\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.524461 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.524384 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-credential-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.524565 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.524459 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-workload-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.524565 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.524490 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-credential-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.524565 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.524540 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-envoy\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.524565 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.524562 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55f64be-7022-43a7-a419-cbf1ab06223a-istiod-ca-cert\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.524808 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.524657 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istiod-ca-cert\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.524808 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.524715 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-podinfo\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.524916 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.524818 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfw99\" (UniqueName: \"kubernetes.io/projected/c55f64be-7022-43a7-a419-cbf1ab06223a-kube-api-access-gfw99\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.524916 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.524873 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-workload-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.525018 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.524923 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-workload-certs\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.525018 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.524942 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llr6v\" (UniqueName: \"kubernetes.io/projected/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-kube-api-access-llr6v\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.525018 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.524991 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-podinfo\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.525176 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.525068 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-workload-certs\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.525176 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.525096 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-token\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.525277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.525173 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-data\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.626379 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.626350 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istiod-ca-cert\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.626500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.626402 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-podinfo\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.626500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.626443 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfw99\" (UniqueName: \"kubernetes.io/projected/c55f64be-7022-43a7-a419-cbf1ab06223a-kube-api-access-gfw99\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.626500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.626494 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-workload-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.626735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.626526 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-workload-certs\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.626735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.626550 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llr6v\" (UniqueName: \"kubernetes.io/projected/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-kube-api-access-llr6v\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.626735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.626588 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-podinfo\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.626735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.626700 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-workload-certs\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.626939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.626740 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-token\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.626939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.626794 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-data\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.626939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.626832 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-data\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.626939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.626865 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-envoy\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.626939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.626891 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-token\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.626939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.626936 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-credential-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.627245 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.626995 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-workload-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.627245 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.627023 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-credential-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.627245 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.627062 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-envoy\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.627245 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.627093 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55f64be-7022-43a7-a419-cbf1ab06223a-istiod-ca-cert\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.627245 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.627117 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istiod-ca-cert\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.627245 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.627189 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-workload-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.627822 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.627793 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-workload-certs\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.628117 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.628092 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-data\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.628202 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.628117 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55f64be-7022-43a7-a419-cbf1ab06223a-istiod-ca-cert\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.628259 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.628238 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-data\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.628385 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.628363 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-credential-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.628585 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.628563 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-workload-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.628823 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.628801 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-credential-socket\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.629058 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.629036 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-workload-certs\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.629634 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.629592 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-podinfo\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.630931 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.630911 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-podinfo\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.631215 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.631189 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-envoy\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.631746 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.631721 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-envoy\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.690940 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.690913 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfw99\" (UniqueName: \"kubernetes.io/projected/c55f64be-7022-43a7-a419-cbf1ab06223a-kube-api-access-gfw99\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.692839 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.692816 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-token\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-lhszx\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:26.693046 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.693030 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llr6v\" (UniqueName: \"kubernetes.io/projected/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-kube-api-access-llr6v\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.727334 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.727306 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-token\") pod \"e2e-worker-w3-cef074-openshift-default-569676869-6bt25\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.945141 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.945071 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:26.953387 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:26.953363 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:27.068260 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:27.068228 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2964de57-6bd4-472c-b74a-4a60b17d7a08" path="/var/lib/kubelet/pods/2964de57-6bd4-472c-b74a-4a60b17d7a08/volumes" Oct 30 22:09:27.068934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:27.068911 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61841cc6-12d1-4821-bc3e-fe1efd7e6763" path="/var/lib/kubelet/pods/61841cc6-12d1-4821-bc3e-fe1efd7e6763/volumes" Oct 30 22:09:27.298352 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:27.298268 4335 generic.go:356] "Generic (PLEG): container finished" podID="de62402c-594f-4a89-910c-434540713997" containerID="15c10858297f400351b61f5f9421652a9225c83c27d7371c0ac4ff6fb5c708f8" exitCode=0 Oct 30 22:09:27.298499 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:27.298364 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6" event={"ID":"de62402c-594f-4a89-910c-434540713997","Type":"ContainerDied","Data":"15c10858297f400351b61f5f9421652a9225c83c27d7371c0ac4ff6fb5c708f8"} Oct 30 22:09:27.393295 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:09:27.393009 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode372cc9b_a3ca_4ad2_975d_cdaa392a87d1.slice/crio-07798d4fd0c4fa852b723b8fcf0c6f0cfa043f5e310c7c0b8307c12a5c4757f5 WatchSource:0}: Error finding container 07798d4fd0c4fa852b723b8fcf0c6f0cfa043f5e310c7c0b8307c12a5c4757f5: Status 404 returned error can't find the container with id 07798d4fd0c4fa852b723b8fcf0c6f0cfa043f5e310c7c0b8307c12a5c4757f5 Oct 30 22:09:27.397675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:27.397632 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:09:27.397789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:27.397727 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:09:27.397789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:27.397772 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:09:27.411791 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:27.411761 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25"] Oct 30 22:09:27.421578 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:27.421546 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx"] Oct 30 22:09:27.437552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:27.437504 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:09:27.437684 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:27.437591 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:09:27.437684 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:27.437654 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:09:27.513419 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:27.513389 4335 scope.go:122] "RemoveContainer" containerID="5920b69bc5bd3646d86cb53ecc511d8546acc238fd0537c25f4d0b03fd9b47a3" Oct 30 22:09:27.526878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:27.526770 4335 scope.go:122] "RemoveContainer" containerID="079cf20760839acc57297b32fce647d3dd31b591b121853ba96969c3563bb9b7" Oct 30 22:09:27.537478 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:27.537460 4335 scope.go:122] "RemoveContainer" containerID="9eaffe32f204bdf513ef221383801424158b43b30e656fd9cc5fee58c5da46aa" Oct 30 22:09:27.549290 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:27.549269 4335 scope.go:122] "RemoveContainer" containerID="21860c68681be48163131b9891364893a73305ac5c6b7d1041fe68e9cce73b57" Oct 30 22:09:27.947724 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:27.947643 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:09:28.012725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.012690 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:09:28.357735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.356762 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" event={"ID":"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1","Type":"ContainerStarted","Data":"de8cd9446ff513c95801d372d3e1dab934ecfcc1ef57b33bccbd20e8bb29018f"} Oct 30 22:09:28.357735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.356832 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" event={"ID":"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1","Type":"ContainerStarted","Data":"07798d4fd0c4fa852b723b8fcf0c6f0cfa043f5e310c7c0b8307c12a5c4757f5"} Oct 30 22:09:28.361363 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.361008 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" event={"ID":"c55f64be-7022-43a7-a419-cbf1ab06223a","Type":"ContainerStarted","Data":"6b183b991dbf27a40f5a78cf78439db5350f3289c03775689637272f8fb9617b"} Oct 30 22:09:28.361363 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.361073 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" event={"ID":"c55f64be-7022-43a7-a419-cbf1ab06223a","Type":"ContainerStarted","Data":"e70d56d6f8819fd4318a0bccee1af826b07fe450f30fc319aeb64fa71f05950d"} Oct 30 22:09:28.574975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.572632 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" podStartSLOduration=2.572597235 podStartE2EDuration="2.572597235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:28.495219585 +0000 UTC m=+1501.595651340" watchObservedRunningTime="2026-10-30 22:09:28.572597235 +0000 UTC m=+1501.673028966" Oct 30 22:09:28.667136 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.667005 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" podStartSLOduration=2.666983401 podStartE2EDuration="2.666983401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:28.633163241 +0000 UTC m=+1501.733594975" watchObservedRunningTime="2026-10-30 22:09:28.666983401 +0000 UTC m=+1501.767415133" Oct 30 22:09:28.773525 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.773499 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6" Oct 30 22:09:28.889139 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.889100 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de62402c-594f-4a89-910c-434540713997-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de62402c-594f-4a89-910c-434540713997-maas-api-openshift-ca\") pod \"de62402c-594f-4a89-910c-434540713997\" (UID: \"de62402c-594f-4a89-910c-434540713997\") " Oct 30 22:09:28.889336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.889319 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de62402c-594f-4a89-910c-434540713997-kube-api-access-dl78m\" (UniqueName: \"kubernetes.io/projected/de62402c-594f-4a89-910c-434540713997-kube-api-access-dl78m\") pod \"de62402c-594f-4a89-910c-434540713997\" (UID: \"de62402c-594f-4a89-910c-434540713997\") " Oct 30 22:09:28.889397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.889382 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de62402c-594f-4a89-910c-434540713997-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de62402c-594f-4a89-910c-434540713997-maas-api-xks-ca\") pod \"de62402c-594f-4a89-910c-434540713997\" (UID: \"de62402c-594f-4a89-910c-434540713997\") " Oct 30 22:09:28.889448 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.889428 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de62402c-594f-4a89-910c-434540713997-maas-api-openshift-ca" pod "de62402c-594f-4a89-910c-434540713997" (UID: "de62402c-594f-4a89-910c-434540713997"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:28.890252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.890216 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de62402c-594f-4a89-910c-434540713997-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:28.891605 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.891577 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de62402c-594f-4a89-910c-434540713997-kube-api-access-dl78m" pod "de62402c-594f-4a89-910c-434540713997" (UID: "de62402c-594f-4a89-910c-434540713997"). InnerVolumeSpecName "kube-api-access-dl78m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:28.891716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.891641 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de62402c-594f-4a89-910c-434540713997-maas-api-xks-ca" pod "de62402c-594f-4a89-910c-434540713997" (UID: "de62402c-594f-4a89-910c-434540713997"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:28.965695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.956965 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:28.965695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.957103 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-lhszx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:09:28.965695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.957194 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 22:09:28.972642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.968226 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-6bt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:09:28.972642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.968284 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 22:09:28.972642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.968357 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:28.992195 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.992162 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl78m\" (UniqueName: \"kubernetes.io/projected/de62402c-594f-4a89-910c-434540713997-kube-api-access-dl78m\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:28.992195 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:28.992190 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de62402c-594f-4a89-910c-434540713997-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:29.079539 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:29.078808 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:09:29.375317 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:29.375277 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6" event={"ID":"de62402c-594f-4a89-910c-434540713997","Type":"ContainerDied","Data":"82a8d2e54ccdd272ede14a2e27d8c520f1d527917e788d1dbfa34f9ebdc3d3ac"} Oct 30 22:09:29.375317 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:29.375303 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6" Oct 30 22:09:29.375542 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:29.375376 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82a8d2e54ccdd272ede14a2e27d8c520f1d527917e788d1dbfa34f9ebdc3d3ac" Oct 30 22:09:29.954639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:29.950019 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-6bt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:09:29.954639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:29.950079 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 22:09:29.963633 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:29.955461 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-lhszx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:09:29.963633 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:29.955507 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 22:09:30.131554 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:30.118710 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp"] Oct 30 22:09:30.195456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:30.195426 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4f728e-5rmrp"] Oct 30 22:09:30.628644 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:30.617895 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:30.706832 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:30.706778 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:30.908931 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:30.908790 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:30.947104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:30.947060 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-6bt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:09:30.947422 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:30.947375 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 22:09:30.958435 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:30.958336 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-lhszx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:09:30.959038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:30.958990 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 22:09:31.005753 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:31.005708 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:31.080640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:31.077710 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:31.101263 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:31.101221 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:31.114640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:31.108102 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f4a2fcd-7972-4f11-a57c-17c8f9c46f51" path="/var/lib/kubelet/pods/8f4a2fcd-7972-4f11-a57c-17c8f9c46f51/volumes" Oct 30 22:09:31.260544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:31.260449 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:31.273728 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:31.273686 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": context deadline exceeded" Oct 30 22:09:31.319670 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:31.319599 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:31.338751 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:31.338701 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:31.391759 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:31.391710 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 22:09:31.417281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:31.417239 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 22:09:31.478824 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:31.478764 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:31.660642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:31.659254 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Oct 30 22:09:31.660642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:31.659709 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:09:31.716760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:31.716705 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:31.717144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:31.717114 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:09:32.099643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.074391 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 30 22:09:32.128829 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.079408 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-6bt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:09:32.129035 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.129010 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 22:09:32.129135 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.104049 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": context deadline exceeded" Oct 30 22:09:32.154633 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.129458 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:09:32.208637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.185060 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:09:32.208637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.102709 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:32.208637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.187070 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:09:32.239640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.239014 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-lhszx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:09:32.239640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.239073 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 22:09:32.293690 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.262366 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:32.314643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.294969 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:09:32.315814 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.269862 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:32.366652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.312269 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:32.366652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.324284 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:09:32.366652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.325234 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:32.366652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.332335 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:09:32.396341 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.394968 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 22:09:32.396341 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.396187 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 22:09:32.425767 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.425712 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:32.427153 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.427127 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:09:32.592193 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.592150 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Oct 30 22:09:32.594543 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.594518 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:09:32.680721 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.660788 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:32.732668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.720703 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:32.986440 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.984992 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-lhszx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:09:32.986440 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.985047 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 22:09:32.998657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.989792 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-6bt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:09:32.998657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:32.989844 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 22:09:33.172537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:33.172496 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:33.301308 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:33.173148 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/health\": context deadline exceeded" Oct 30 22:09:33.301537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:33.173595 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:33.302247 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:33.217764 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:33.320640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:33.297239 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:33.321009 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:33.320974 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": context deadline exceeded" Oct 30 22:09:33.348642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:33.323736 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Oct 30 22:09:33.349213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:33.344652 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": context deadline exceeded" Oct 30 22:09:33.433643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:33.409417 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:33.459641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:33.438740 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:33.576696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:33.535009 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:09:33.716529 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:33.601699 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Oct 30 22:09:33.717147 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:33.667273 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:33.742946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:33.742888 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:33.821642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:33.797271 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded" Oct 30 22:09:33.997737 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:33.986781 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-lhszx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:09:33.997737 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:33.986837 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 22:09:34.008732 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:34.008688 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-6bt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:09:34.013658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:34.008997 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 22:09:34.304457 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:34.304407 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": context deadline exceeded" Oct 30 22:09:34.305928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:34.305896 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:34.315663 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:34.308706 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 30 22:09:34.348673 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:34.337759 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:34.348673 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:34.338069 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 22:09:34.351060 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:34.350462 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:34.354778 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:34.350603 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Oct 30 22:09:34.779702 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:34.540589 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 22:09:34.779702 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:34.771931 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:34.779702 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:34.772980 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Oct 30 22:09:34.788703 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:34.783857 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 30 22:09:35.014644 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:34.977481 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-lhszx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:09:35.014644 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:34.977583 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 22:09:35.048963 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.048916 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": context deadline exceeded" Oct 30 22:09:35.320689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.320566 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:35.387683 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.321325 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 30 22:09:35.391711 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.346588 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-6bt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:09:35.391997 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.391948 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 22:09:35.416644 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.372949 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:35.417829 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.404004 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": context deadline exceeded" Oct 30 22:09:35.418525 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.404121 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 22:09:35.484642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.477499 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Oct 30 22:09:35.643695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.509580 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": context deadline exceeded" Oct 30 22:09:35.806867 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.806819 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 30 22:09:35.816858 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.816706 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:35.820049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.819998 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:35.820339 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.820310 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Oct 30 22:09:35.937812 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.936782 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-c9x78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:09:35.937812 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.936851 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" podUID="d6978eea-f24f-4d30-83b0-85b0015baf53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:35.992642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.967700 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-lhszx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:09:35.992642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.967762 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 22:09:35.993212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.993177 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-6bt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:09:35.993387 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:35.993360 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 22:09:36.022118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.002204 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 22:09:36.285962 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.285863 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-tpwc7" podUID="3e3c54c2-eb66-4b04-b6ef-55c1691873b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:36.386955 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.386899 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:36.426439 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.426381 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:36.457727 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.457663 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 22:09:36.727794 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.458713 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:36.728394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.458746 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 22:09:36.739650 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.541990 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Oct 30 22:09:36.739650 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.607437 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/health\": context deadline exceeded" Oct 30 22:09:36.739650 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:09:36.657476 4335 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.527s" Oct 30 22:09:36.739650 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.696458 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": context deadline exceeded" Oct 30 22:09:36.838637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.822783 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:36.838637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.823779 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 30 22:09:36.838880 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.838689 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:36.838880 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.838782 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Oct 30 22:09:36.890991 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.824946 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:36.988705 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.961769 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-lhszx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:09:36.988705 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.961828 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 22:09:36.993862 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.993827 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-6bt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:09:36.994101 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:36.994072 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 22:09:37.022334 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:37.020820 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:37.052847 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:37.052794 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:09:37.053148 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:37.053105 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" podUID="769e08d0-bf5e-404e-8b7a-adf43dcd0905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:37.357763 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:37.357666 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/health\": context deadline exceeded" Oct 30 22:09:37.392442 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:37.392396 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:09:37.401549 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:37.396781 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" podUID="0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:37.431641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:37.424358 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:37.604809 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:37.450708 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:37.661235 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:37.661136 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:09:37.661528 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:37.661470 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" podUID="6c2f07ba-2e3c-45b2-b619-7ce607083590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:37.706582 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:37.706526 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:37.734772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:37.734731 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:37.735098 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:37.735070 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:37.735325 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:37.735302 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:37.735516 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:37.735480 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 22:09:37.825782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:37.825735 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:37.827221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:37.827122 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 30 22:09:37.856994 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:37.856954 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Oct 30 22:09:38.004402 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:38.004305 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-6bt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:09:38.004816 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:38.004695 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 22:09:38.007936 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:38.007906 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded" Oct 30 22:09:38.008117 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:38.008096 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 22:09:38.040741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:38.030548 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:38.117306 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:38.117272 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-lhszx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:09:38.117570 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:38.117535 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 22:09:38.227670 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:38.202666 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh"] Oct 30 22:09:38.267529 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:38.267434 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:09:38.271104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:38.271052 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" podUID="a3b3ae09-c03e-4c8a-95ea-31140419aa9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:38.277659 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:38.273270 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0c1780d-c8xsh"] Oct 30 22:09:38.740437 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:38.740392 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:09:38.871134 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:38.871090 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:38.996175 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:38.983465 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 30 22:09:39.139776 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:38.918744 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Oct 30 22:09:39.158887 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:39.155932 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:39.192376 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:39.192162 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 22:09:39.245449 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:39.245384 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-lhszx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:09:39.245788 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:39.245755 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 22:09:39.262790 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:39.262697 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:39.385636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:39.371220 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:39.385636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:39.373016 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-6bt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:09:39.385636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:39.373058 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 22:09:39.385636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:39.374274 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 22:09:39.393243 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:38.893994 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:39.393830 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:39.393792 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": context deadline exceeded" Oct 30 22:09:39.394274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:39.394228 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": context deadline exceeded" Oct 30 22:09:39.394735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:39.394712 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 22:09:39.395160 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:39.395140 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/health\": context deadline exceeded" Oct 30 22:09:39.527703 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:39.513363 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:09:39.695140 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:39.695096 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf66bbd9-61d9-432c-9e5b-a525eb60945a" path="/var/lib/kubelet/pods/bf66bbd9-61d9-432c-9e5b-a525eb60945a/volumes" Oct 30 22:09:39.699643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:39.695706 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 22:09:39.699643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:39.695986 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:09:39.948285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:39.948239 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:40.049091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.047750 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-lhszx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:09:40.049091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.047901 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 22:09:40.049091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.048087 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-6bt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:09:40.049091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.048214 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 22:09:40.089834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.089807 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:09:40.095051 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.095021 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d"] Oct 30 22:09:40.099786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.099771 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:09:40.114714 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.114683 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:40.172784 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.172739 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2be64e-7vj4d"] Oct 30 22:09:40.195175 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.195141 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:40.288693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.286888 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:09:40.353640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.345342 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:09:40.397678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.395498 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:40.397678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.396594 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": context deadline exceeded" Oct 30 22:09:40.424698 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.420721 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:09:40.477855 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.477826 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:09:40.482421 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.482398 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:09:40.576185 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.576158 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:09:40.948149 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.948068 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-6bt25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 22:09:40.948397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.948364 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 22:09:40.953760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.953713 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-lhszx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:09:40.953869 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.953778 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 22:09:40.963292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.963266 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5"] Oct 30 22:09:40.964404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.964387 4335 cpu_manager.go:414] "containerMap: removing container" podUID="de62402c-594f-4a89-910c-434540713997" containerName="revoke-keys" Oct 30 22:09:40.964522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.964510 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de62402c-594f-4a89-910c-434540713997" containerName="revoke-keys" Oct 30 22:09:40.964838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.964824 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="de62402c-594f-4a89-910c-434540713997" containerName="revoke-keys" Oct 30 22:09:40.970924 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:40.970905 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5" Oct 30 22:09:41.015385 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.015288 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5"] Oct 30 22:09:41.030478 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.030451 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:41.033821 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.030810 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:09:41.038564 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.033711 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:41.038564 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.033804 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:41.073659 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.071890 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4b181a-794b-4f14-8091-f0e821da1bdb" path="/var/lib/kubelet/pods/7b4b181a-794b-4f14-8091-f0e821da1bdb/volumes" Oct 30 22:09:41.087034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.087002 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98131f4-7440-4740-b286-3dea16a9f939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eeb3b5f-mkvb5\" (UID: \"f98131f4-7440-4740-b286-3dea16a9f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5" Oct 30 22:09:41.087292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.087272 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98131f4-7440-4740-b286-3dea16a9f939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eeb3b5f-mkvb5\" (UID: \"f98131f4-7440-4740-b286-3dea16a9f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5" Oct 30 22:09:41.087746 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.087726 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jjhz\" (UniqueName: \"kubernetes.io/projected/f98131f4-7440-4740-b286-3dea16a9f939-kube-api-access-4jjhz\") pod \"maas-api-revoke-sub-0eeb3b5f-mkvb5\" (UID: \"f98131f4-7440-4740-b286-3dea16a9f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5" Oct 30 22:09:41.189775 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.189742 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jjhz\" (UniqueName: \"kubernetes.io/projected/f98131f4-7440-4740-b286-3dea16a9f939-kube-api-access-4jjhz\") pod \"maas-api-revoke-sub-0eeb3b5f-mkvb5\" (UID: \"f98131f4-7440-4740-b286-3dea16a9f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5" Oct 30 22:09:41.190057 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.190030 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98131f4-7440-4740-b286-3dea16a9f939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eeb3b5f-mkvb5\" (UID: \"f98131f4-7440-4740-b286-3dea16a9f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5" Oct 30 22:09:41.190215 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.190198 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98131f4-7440-4740-b286-3dea16a9f939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eeb3b5f-mkvb5\" (UID: \"f98131f4-7440-4740-b286-3dea16a9f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5" Oct 30 22:09:41.191107 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.190856 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98131f4-7440-4740-b286-3dea16a9f939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eeb3b5f-mkvb5\" (UID: \"f98131f4-7440-4740-b286-3dea16a9f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5" Oct 30 22:09:41.193530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.193506 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98131f4-7440-4740-b286-3dea16a9f939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eeb3b5f-mkvb5\" (UID: \"f98131f4-7440-4740-b286-3dea16a9f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5" Oct 30 22:09:41.251298 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.251231 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jjhz\" (UniqueName: \"kubernetes.io/projected/f98131f4-7440-4740-b286-3dea16a9f939-kube-api-access-4jjhz\") pod \"maas-api-revoke-sub-0eeb3b5f-mkvb5\" (UID: \"f98131f4-7440-4740-b286-3dea16a9f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5" Oct 30 22:09:41.308900 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.308680 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5" Oct 30 22:09:41.749990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.749960 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5"] Oct 30 22:09:41.946666 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.946223 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:41.954165 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:41.954140 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:42.339838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.339429 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5" event={"ID":"f98131f4-7440-4740-b286-3dea16a9f939","Type":"ContainerStarted","Data":"547ac3bfb5028c2ded88a3a43540e9d8e32abd32d30b9dcf364d928d761ed231"} Oct 30 22:09:42.339838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.339530 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5" event={"ID":"f98131f4-7440-4740-b286-3dea16a9f939","Type":"ContainerStarted","Data":"71226058357bf7047487313e3494715e667b171e25fc7571fe29775b37374e1a"} Oct 30 22:09:42.405486 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.405407 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk"] Oct 30 22:09:42.420909 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.420884 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk" Oct 30 22:09:42.459227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.459189 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlzmv\" (UniqueName: \"kubernetes.io/projected/41189525-2831-42f8-bcec-8426752c71e8-kube-api-access-qlzmv\") pod \"maas-api-revoke-sub-2d92705e-mghsk\" (UID: \"41189525-2831-42f8-bcec-8426752c71e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk" Oct 30 22:09:42.459409 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.459264 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41189525-2831-42f8-bcec-8426752c71e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d92705e-mghsk\" (UID: \"41189525-2831-42f8-bcec-8426752c71e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk" Oct 30 22:09:42.459409 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.459378 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41189525-2831-42f8-bcec-8426752c71e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d92705e-mghsk\" (UID: \"41189525-2831-42f8-bcec-8426752c71e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk" Oct 30 22:09:42.461329 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.461295 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk"] Oct 30 22:09:42.466150 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.466107 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5" podStartSLOduration=2.466093805 podStartE2EDuration="2.466093805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:42.464122077 +0000 UTC m=+1515.564553809" watchObservedRunningTime="2026-10-30 22:09:42.466093805 +0000 UTC m=+1515.566525537" Oct 30 22:09:42.481675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.481647 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:42.484652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.483955 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:09:42.535005 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.532562 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:42.535944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.535922 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:09:42.561872 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.561844 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlzmv\" (UniqueName: \"kubernetes.io/projected/41189525-2831-42f8-bcec-8426752c71e8-kube-api-access-qlzmv\") pod \"maas-api-revoke-sub-2d92705e-mghsk\" (UID: \"41189525-2831-42f8-bcec-8426752c71e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk" Oct 30 22:09:42.562136 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.562117 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41189525-2831-42f8-bcec-8426752c71e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d92705e-mghsk\" (UID: \"41189525-2831-42f8-bcec-8426752c71e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk" Oct 30 22:09:42.562363 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.562347 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41189525-2831-42f8-bcec-8426752c71e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d92705e-mghsk\" (UID: \"41189525-2831-42f8-bcec-8426752c71e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk" Oct 30 22:09:42.563173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.563152 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41189525-2831-42f8-bcec-8426752c71e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d92705e-mghsk\" (UID: \"41189525-2831-42f8-bcec-8426752c71e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk" Oct 30 22:09:42.568377 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.568357 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41189525-2831-42f8-bcec-8426752c71e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d92705e-mghsk\" (UID: \"41189525-2831-42f8-bcec-8426752c71e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk" Oct 30 22:09:42.623451 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.623416 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlzmv\" (UniqueName: \"kubernetes.io/projected/41189525-2831-42f8-bcec-8426752c71e8-kube-api-access-qlzmv\") pod \"maas-api-revoke-sub-2d92705e-mghsk\" (UID: \"41189525-2831-42f8-bcec-8426752c71e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk" Oct 30 22:09:42.751320 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:42.751204 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk" Oct 30 22:09:43.232921 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:43.228222 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk"] Oct 30 22:09:43.366552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:43.366520 4335 generic.go:356] "Generic (PLEG): container finished" podID="f98131f4-7440-4740-b286-3dea16a9f939" containerID="547ac3bfb5028c2ded88a3a43540e9d8e32abd32d30b9dcf364d928d761ed231" exitCode=0 Oct 30 22:09:43.367052 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:43.367024 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5" event={"ID":"f98131f4-7440-4740-b286-3dea16a9f939","Type":"ContainerDied","Data":"547ac3bfb5028c2ded88a3a43540e9d8e32abd32d30b9dcf364d928d761ed231"} Oct 30 22:09:43.373035 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:43.373005 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk" event={"ID":"41189525-2831-42f8-bcec-8426752c71e8","Type":"ContainerStarted","Data":"0e1c3e1e1a0af99aaf4e9596b6bd84b04da3ec95c47fa613bdb4d5a79cc7c2b4"} Oct 30 22:09:44.391415 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:44.390970 4335 generic.go:356] "Generic (PLEG): container finished" podID="41189525-2831-42f8-bcec-8426752c71e8" containerID="609ca12fc43812ddb27cd394d798b266456b1dedf14c6b76a291fd4b5b17c3e3" exitCode=0 Oct 30 22:09:44.391415 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:44.391009 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk" event={"ID":"41189525-2831-42f8-bcec-8426752c71e8","Type":"ContainerDied","Data":"609ca12fc43812ddb27cd394d798b266456b1dedf14c6b76a291fd4b5b17c3e3"} Oct 30 22:09:44.574321 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:44.574298 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5" Oct 30 22:09:44.602840 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:44.602816 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f98131f4-7440-4740-b286-3dea16a9f939-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98131f4-7440-4740-b286-3dea16a9f939-maas-api-xks-ca\") pod \"f98131f4-7440-4740-b286-3dea16a9f939\" (UID: \"f98131f4-7440-4740-b286-3dea16a9f939\") " Oct 30 22:09:44.602973 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:44.602891 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f98131f4-7440-4740-b286-3dea16a9f939-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98131f4-7440-4740-b286-3dea16a9f939-maas-api-openshift-ca\") pod \"f98131f4-7440-4740-b286-3dea16a9f939\" (UID: \"f98131f4-7440-4740-b286-3dea16a9f939\") " Oct 30 22:09:44.602973 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:44.602931 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98131f4-7440-4740-b286-3dea16a9f939-kube-api-access-4jjhz\" (UniqueName: \"kubernetes.io/projected/f98131f4-7440-4740-b286-3dea16a9f939-kube-api-access-4jjhz\") pod \"f98131f4-7440-4740-b286-3dea16a9f939\" (UID: \"f98131f4-7440-4740-b286-3dea16a9f939\") " Oct 30 22:09:44.603278 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:44.603252 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98131f4-7440-4740-b286-3dea16a9f939-maas-api-openshift-ca" pod "f98131f4-7440-4740-b286-3dea16a9f939" (UID: "f98131f4-7440-4740-b286-3dea16a9f939"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:44.604022 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:44.603994 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98131f4-7440-4740-b286-3dea16a9f939-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:44.605084 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:44.605046 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f98131f4-7440-4740-b286-3dea16a9f939-maas-api-xks-ca" pod "f98131f4-7440-4740-b286-3dea16a9f939" (UID: "f98131f4-7440-4740-b286-3dea16a9f939"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:44.605322 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:44.605302 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98131f4-7440-4740-b286-3dea16a9f939-kube-api-access-4jjhz" pod "f98131f4-7440-4740-b286-3dea16a9f939" (UID: "f98131f4-7440-4740-b286-3dea16a9f939"). InnerVolumeSpecName "kube-api-access-4jjhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:44.706120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:44.706089 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jjhz\" (UniqueName: \"kubernetes.io/projected/f98131f4-7440-4740-b286-3dea16a9f939-kube-api-access-4jjhz\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:44.706120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:44.706123 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98131f4-7440-4740-b286-3dea16a9f939-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:45.402988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:45.402910 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5" Oct 30 22:09:45.402988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:45.402950 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5" event={"ID":"f98131f4-7440-4740-b286-3dea16a9f939","Type":"ContainerDied","Data":"71226058357bf7047487313e3494715e667b171e25fc7571fe29775b37374e1a"} Oct 30 22:09:45.403411 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:45.403005 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71226058357bf7047487313e3494715e667b171e25fc7571fe29775b37374e1a" Oct 30 22:09:45.594607 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:45.594581 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk" Oct 30 22:09:45.718896 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:45.718805 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41189525-2831-42f8-bcec-8426752c71e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41189525-2831-42f8-bcec-8426752c71e8-maas-api-openshift-ca\") pod \"41189525-2831-42f8-bcec-8426752c71e8\" (UID: \"41189525-2831-42f8-bcec-8426752c71e8\") " Oct 30 22:09:45.719293 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:45.718929 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41189525-2831-42f8-bcec-8426752c71e8-kube-api-access-qlzmv\" (UniqueName: \"kubernetes.io/projected/41189525-2831-42f8-bcec-8426752c71e8-kube-api-access-qlzmv\") pod \"41189525-2831-42f8-bcec-8426752c71e8\" (UID: \"41189525-2831-42f8-bcec-8426752c71e8\") " Oct 30 22:09:45.719293 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:45.719062 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41189525-2831-42f8-bcec-8426752c71e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41189525-2831-42f8-bcec-8426752c71e8-maas-api-xks-ca\") pod \"41189525-2831-42f8-bcec-8426752c71e8\" (UID: \"41189525-2831-42f8-bcec-8426752c71e8\") " Oct 30 22:09:45.719293 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:45.719187 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41189525-2831-42f8-bcec-8426752c71e8-maas-api-openshift-ca" pod "41189525-2831-42f8-bcec-8426752c71e8" (UID: "41189525-2831-42f8-bcec-8426752c71e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:45.719535 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:45.719517 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41189525-2831-42f8-bcec-8426752c71e8-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:45.721158 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:45.721132 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41189525-2831-42f8-bcec-8426752c71e8-kube-api-access-qlzmv" pod "41189525-2831-42f8-bcec-8426752c71e8" (UID: "41189525-2831-42f8-bcec-8426752c71e8"). InnerVolumeSpecName "kube-api-access-qlzmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:45.721341 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:45.721210 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41189525-2831-42f8-bcec-8426752c71e8-maas-api-xks-ca" pod "41189525-2831-42f8-bcec-8426752c71e8" (UID: "41189525-2831-42f8-bcec-8426752c71e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:45.821244 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:45.821217 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlzmv\" (UniqueName: \"kubernetes.io/projected/41189525-2831-42f8-bcec-8426752c71e8-kube-api-access-qlzmv\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:45.821244 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:45.821241 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41189525-2831-42f8-bcec-8426752c71e8-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:46.415534 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:46.415499 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk" Oct 30 22:09:46.415926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:46.415529 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk" event={"ID":"41189525-2831-42f8-bcec-8426752c71e8","Type":"ContainerDied","Data":"0e1c3e1e1a0af99aaf4e9596b6bd84b04da3ec95c47fa613bdb4d5a79cc7c2b4"} Oct 30 22:09:46.415926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:46.415585 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e1c3e1e1a0af99aaf4e9596b6bd84b04da3ec95c47fa613bdb4d5a79cc7c2b4" Oct 30 22:09:47.139783 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.139753 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq"] Oct 30 22:09:47.140364 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.140352 4335 cpu_manager.go:414] "containerMap: removing container" podUID="41189525-2831-42f8-bcec-8426752c71e8" containerName="revoke-keys" Oct 30 22:09:47.140410 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.140366 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41189525-2831-42f8-bcec-8426752c71e8" containerName="revoke-keys" Oct 30 22:09:47.140410 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.140386 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f98131f4-7440-4740-b286-3dea16a9f939" containerName="revoke-keys" Oct 30 22:09:47.140410 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.140391 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98131f4-7440-4740-b286-3dea16a9f939" containerName="revoke-keys" Oct 30 22:09:47.140516 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.140507 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="41189525-2831-42f8-bcec-8426752c71e8" containerName="revoke-keys" Oct 30 22:09:47.140554 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.140519 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98131f4-7440-4740-b286-3dea16a9f939" containerName="revoke-keys" Oct 30 22:09:47.144017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.143997 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq" Oct 30 22:09:47.201667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.201642 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq"] Oct 30 22:09:47.240049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.240017 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7tr5\" (UniqueName: \"kubernetes.io/projected/49045d81-6fce-40bf-8adc-8bffaaeefbc6-kube-api-access-z7tr5\") pod \"maas-api-revoke-sub-54ced5c2-277wq\" (UID: \"49045d81-6fce-40bf-8adc-8bffaaeefbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq" Oct 30 22:09:47.240215 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.240094 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49045d81-6fce-40bf-8adc-8bffaaeefbc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54ced5c2-277wq\" (UID: \"49045d81-6fce-40bf-8adc-8bffaaeefbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq" Oct 30 22:09:47.240381 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.240360 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49045d81-6fce-40bf-8adc-8bffaaeefbc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54ced5c2-277wq\" (UID: \"49045d81-6fce-40bf-8adc-8bffaaeefbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq" Oct 30 22:09:47.244453 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.244428 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:47.244572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.244482 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:09:47.244572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.244533 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:47.244898 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.244874 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:47.341514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.341493 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49045d81-6fce-40bf-8adc-8bffaaeefbc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54ced5c2-277wq\" (UID: \"49045d81-6fce-40bf-8adc-8bffaaeefbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq" Oct 30 22:09:47.341673 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.341567 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7tr5\" (UniqueName: \"kubernetes.io/projected/49045d81-6fce-40bf-8adc-8bffaaeefbc6-kube-api-access-z7tr5\") pod \"maas-api-revoke-sub-54ced5c2-277wq\" (UID: \"49045d81-6fce-40bf-8adc-8bffaaeefbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq" Oct 30 22:09:47.341673 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.341599 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49045d81-6fce-40bf-8adc-8bffaaeefbc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54ced5c2-277wq\" (UID: \"49045d81-6fce-40bf-8adc-8bffaaeefbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq" Oct 30 22:09:47.342293 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.342273 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49045d81-6fce-40bf-8adc-8bffaaeefbc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54ced5c2-277wq\" (UID: \"49045d81-6fce-40bf-8adc-8bffaaeefbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq" Oct 30 22:09:47.344401 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.344380 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49045d81-6fce-40bf-8adc-8bffaaeefbc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54ced5c2-277wq\" (UID: \"49045d81-6fce-40bf-8adc-8bffaaeefbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq" Oct 30 22:09:47.402467 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.402381 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7tr5\" (UniqueName: \"kubernetes.io/projected/49045d81-6fce-40bf-8adc-8bffaaeefbc6-kube-api-access-z7tr5\") pod \"maas-api-revoke-sub-54ced5c2-277wq\" (UID: \"49045d81-6fce-40bf-8adc-8bffaaeefbc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq" Oct 30 22:09:47.458884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.458856 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq" Oct 30 22:09:47.842633 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:09:47.842595 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49045d81_6fce_40bf_8adc_8bffaaeefbc6.slice/crio-4630d80c138cce728a3d313658545c85539204ca8b1a4e5977fd6656f4c18f44 WatchSource:0}: Error finding container 4630d80c138cce728a3d313658545c85539204ca8b1a4e5977fd6656f4c18f44: Status 404 returned error can't find the container with id 4630d80c138cce728a3d313658545c85539204ca8b1a4e5977fd6656f4c18f44 Oct 30 22:09:47.844866 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:47.844836 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq"] Oct 30 22:09:48.034698 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:48.034674 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s"] Oct 30 22:09:48.040438 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:48.040419 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s" Oct 30 22:09:48.102667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:48.102240 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s"] Oct 30 22:09:48.153119 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:48.153083 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45ce109-ed30-459e-954f-452d1ab9e7c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4031e7a9-k797s\" (UID: \"d45ce109-ed30-459e-954f-452d1ab9e7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s" Oct 30 22:09:48.153315 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:48.153214 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnvbb\" (UniqueName: \"kubernetes.io/projected/d45ce109-ed30-459e-954f-452d1ab9e7c4-kube-api-access-fnvbb\") pod \"maas-api-revoke-sub-4031e7a9-k797s\" (UID: \"d45ce109-ed30-459e-954f-452d1ab9e7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s" Oct 30 22:09:48.153428 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:48.153409 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45ce109-ed30-459e-954f-452d1ab9e7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4031e7a9-k797s\" (UID: \"d45ce109-ed30-459e-954f-452d1ab9e7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s" Oct 30 22:09:48.254654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:48.254606 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45ce109-ed30-459e-954f-452d1ab9e7c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4031e7a9-k797s\" (UID: \"d45ce109-ed30-459e-954f-452d1ab9e7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s" Oct 30 22:09:48.254760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:48.254722 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnvbb\" (UniqueName: \"kubernetes.io/projected/d45ce109-ed30-459e-954f-452d1ab9e7c4-kube-api-access-fnvbb\") pod \"maas-api-revoke-sub-4031e7a9-k797s\" (UID: \"d45ce109-ed30-459e-954f-452d1ab9e7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s" Oct 30 22:09:48.254880 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:48.254863 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45ce109-ed30-459e-954f-452d1ab9e7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4031e7a9-k797s\" (UID: \"d45ce109-ed30-459e-954f-452d1ab9e7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s" Oct 30 22:09:48.266773 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:48.266743 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45ce109-ed30-459e-954f-452d1ab9e7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4031e7a9-k797s\" (UID: \"d45ce109-ed30-459e-954f-452d1ab9e7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s" Oct 30 22:09:48.266916 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:48.266897 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45ce109-ed30-459e-954f-452d1ab9e7c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4031e7a9-k797s\" (UID: \"d45ce109-ed30-459e-954f-452d1ab9e7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s" Oct 30 22:09:48.316226 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:48.316207 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnvbb\" (UniqueName: \"kubernetes.io/projected/d45ce109-ed30-459e-954f-452d1ab9e7c4-kube-api-access-fnvbb\") pod \"maas-api-revoke-sub-4031e7a9-k797s\" (UID: \"d45ce109-ed30-459e-954f-452d1ab9e7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s" Oct 30 22:09:48.354029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:48.353970 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s" Oct 30 22:09:48.444315 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:48.444279 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq" event={"ID":"49045d81-6fce-40bf-8adc-8bffaaeefbc6","Type":"ContainerStarted","Data":"674050046b40ae24c4a90dc8a64e50663b7ae4e73c9d5bd02ebbe968abc149d3"} Oct 30 22:09:48.444464 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:48.444377 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq" event={"ID":"49045d81-6fce-40bf-8adc-8bffaaeefbc6","Type":"ContainerStarted","Data":"4630d80c138cce728a3d313658545c85539204ca8b1a4e5977fd6656f4c18f44"} Oct 30 22:09:48.563541 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:48.563485 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq" podStartSLOduration=1.5634706409999999 podStartE2EDuration="1.563470641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:48.560653566 +0000 UTC m=+1521.661085298" watchObservedRunningTime="2026-10-30 22:09:48.563470641 +0000 UTC m=+1521.663902372" Oct 30 22:09:48.746590 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:09:48.746553 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45ce109_ed30_459e_954f_452d1ab9e7c4.slice/crio-6980f0322037785440e025e10eb47e9f8c7049e0e9b3690f7ec64700a3ff43db WatchSource:0}: Error finding container 6980f0322037785440e025e10eb47e9f8c7049e0e9b3690f7ec64700a3ff43db: Status 404 returned error can't find the container with id 6980f0322037785440e025e10eb47e9f8c7049e0e9b3690f7ec64700a3ff43db Oct 30 22:09:48.747504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:48.747372 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s"] Oct 30 22:09:49.454801 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:49.454769 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s" event={"ID":"d45ce109-ed30-459e-954f-452d1ab9e7c4","Type":"ContainerStarted","Data":"b5aac2395adde8b737f884f46ea6e4c8bce9a1ac4e4526ce18b819ba1d94a997"} Oct 30 22:09:49.454977 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:49.454835 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s" event={"ID":"d45ce109-ed30-459e-954f-452d1ab9e7c4","Type":"ContainerStarted","Data":"6980f0322037785440e025e10eb47e9f8c7049e0e9b3690f7ec64700a3ff43db"} Oct 30 22:09:49.456456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:49.456432 4335 generic.go:356] "Generic (PLEG): container finished" podID="49045d81-6fce-40bf-8adc-8bffaaeefbc6" containerID="674050046b40ae24c4a90dc8a64e50663b7ae4e73c9d5bd02ebbe968abc149d3" exitCode=0 Oct 30 22:09:49.456580 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:49.456517 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq" event={"ID":"49045d81-6fce-40bf-8adc-8bffaaeefbc6","Type":"ContainerDied","Data":"674050046b40ae24c4a90dc8a64e50663b7ae4e73c9d5bd02ebbe968abc149d3"} Oct 30 22:09:49.578264 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:49.578216 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s" podStartSLOduration=2.578203253 podStartE2EDuration="2.578203253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:49.572328121 +0000 UTC m=+1522.672759855" watchObservedRunningTime="2026-10-30 22:09:49.578203253 +0000 UTC m=+1522.678634993" Oct 30 22:09:50.466654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:50.466601 4335 generic.go:356] "Generic (PLEG): container finished" podID="d45ce109-ed30-459e-954f-452d1ab9e7c4" containerID="b5aac2395adde8b737f884f46ea6e4c8bce9a1ac4e4526ce18b819ba1d94a997" exitCode=0 Oct 30 22:09:50.466826 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:50.466689 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s" event={"ID":"d45ce109-ed30-459e-954f-452d1ab9e7c4","Type":"ContainerDied","Data":"b5aac2395adde8b737f884f46ea6e4c8bce9a1ac4e4526ce18b819ba1d94a997"} Oct 30 22:09:50.642070 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:50.642044 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq" Oct 30 22:09:50.788596 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:50.788517 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49045d81-6fce-40bf-8adc-8bffaaeefbc6-kube-api-access-z7tr5\" (UniqueName: \"kubernetes.io/projected/49045d81-6fce-40bf-8adc-8bffaaeefbc6-kube-api-access-z7tr5\") pod \"49045d81-6fce-40bf-8adc-8bffaaeefbc6\" (UID: \"49045d81-6fce-40bf-8adc-8bffaaeefbc6\") " Oct 30 22:09:50.788806 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:50.788784 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49045d81-6fce-40bf-8adc-8bffaaeefbc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49045d81-6fce-40bf-8adc-8bffaaeefbc6-maas-api-xks-ca\") pod \"49045d81-6fce-40bf-8adc-8bffaaeefbc6\" (UID: \"49045d81-6fce-40bf-8adc-8bffaaeefbc6\") " Oct 30 22:09:50.788969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:50.788945 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49045d81-6fce-40bf-8adc-8bffaaeefbc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49045d81-6fce-40bf-8adc-8bffaaeefbc6-maas-api-openshift-ca\") pod \"49045d81-6fce-40bf-8adc-8bffaaeefbc6\" (UID: \"49045d81-6fce-40bf-8adc-8bffaaeefbc6\") " Oct 30 22:09:50.789349 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:50.789322 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49045d81-6fce-40bf-8adc-8bffaaeefbc6-maas-api-openshift-ca" pod "49045d81-6fce-40bf-8adc-8bffaaeefbc6" (UID: "49045d81-6fce-40bf-8adc-8bffaaeefbc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:50.789837 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:50.789813 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49045d81-6fce-40bf-8adc-8bffaaeefbc6-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:50.791024 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:50.790996 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49045d81-6fce-40bf-8adc-8bffaaeefbc6-kube-api-access-z7tr5" pod "49045d81-6fce-40bf-8adc-8bffaaeefbc6" (UID: "49045d81-6fce-40bf-8adc-8bffaaeefbc6"). InnerVolumeSpecName "kube-api-access-z7tr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:50.791188 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:50.791167 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49045d81-6fce-40bf-8adc-8bffaaeefbc6-maas-api-xks-ca" pod "49045d81-6fce-40bf-8adc-8bffaaeefbc6" (UID: "49045d81-6fce-40bf-8adc-8bffaaeefbc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:50.891690 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:50.891664 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7tr5\" (UniqueName: \"kubernetes.io/projected/49045d81-6fce-40bf-8adc-8bffaaeefbc6-kube-api-access-z7tr5\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:50.891794 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:50.891692 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49045d81-6fce-40bf-8adc-8bffaaeefbc6-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:51.480028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.479942 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq" Oct 30 22:09:51.480028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.479968 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq" event={"ID":"49045d81-6fce-40bf-8adc-8bffaaeefbc6","Type":"ContainerDied","Data":"4630d80c138cce728a3d313658545c85539204ca8b1a4e5977fd6656f4c18f44"} Oct 30 22:09:51.480028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.480026 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4630d80c138cce728a3d313658545c85539204ca8b1a4e5977fd6656f4c18f44" Oct 30 22:09:51.652064 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.652041 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s" Oct 30 22:09:51.695439 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.695419 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7"] Oct 30 22:09:51.696169 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.696153 4335 cpu_manager.go:414] "containerMap: removing container" podUID="49045d81-6fce-40bf-8adc-8bffaaeefbc6" containerName="revoke-keys" Oct 30 22:09:51.696221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.696172 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49045d81-6fce-40bf-8adc-8bffaaeefbc6" containerName="revoke-keys" Oct 30 22:09:51.696221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.696203 4335 cpu_manager.go:414] "containerMap: removing container" podUID="d45ce109-ed30-459e-954f-452d1ab9e7c4" containerName="revoke-keys" Oct 30 22:09:51.696221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.696208 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d45ce109-ed30-459e-954f-452d1ab9e7c4" containerName="revoke-keys" Oct 30 22:09:51.696334 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.696309 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="d45ce109-ed30-459e-954f-452d1ab9e7c4" containerName="revoke-keys" Oct 30 22:09:51.696334 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.696325 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="49045d81-6fce-40bf-8adc-8bffaaeefbc6" containerName="revoke-keys" Oct 30 22:09:51.700023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.700007 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7" Oct 30 22:09:51.757035 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.756979 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7"] Oct 30 22:09:51.801438 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.801407 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d45ce109-ed30-459e-954f-452d1ab9e7c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45ce109-ed30-459e-954f-452d1ab9e7c4-maas-api-xks-ca\") pod \"d45ce109-ed30-459e-954f-452d1ab9e7c4\" (UID: \"d45ce109-ed30-459e-954f-452d1ab9e7c4\") " Oct 30 22:09:51.801593 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.801547 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d45ce109-ed30-459e-954f-452d1ab9e7c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45ce109-ed30-459e-954f-452d1ab9e7c4-maas-api-openshift-ca\") pod \"d45ce109-ed30-459e-954f-452d1ab9e7c4\" (UID: \"d45ce109-ed30-459e-954f-452d1ab9e7c4\") " Oct 30 22:09:51.801823 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.801801 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d45ce109-ed30-459e-954f-452d1ab9e7c4-kube-api-access-fnvbb\" (UniqueName: \"kubernetes.io/projected/d45ce109-ed30-459e-954f-452d1ab9e7c4-kube-api-access-fnvbb\") pod \"d45ce109-ed30-459e-954f-452d1ab9e7c4\" (UID: \"d45ce109-ed30-459e-954f-452d1ab9e7c4\") " Oct 30 22:09:51.802329 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.802304 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a255089b-e277-48a1-90b0-043db61a9443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98daec9f-hfcn7\" (UID: \"a255089b-e277-48a1-90b0-043db61a9443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7" Oct 30 22:09:51.802657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.802408 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d45ce109-ed30-459e-954f-452d1ab9e7c4-maas-api-openshift-ca" pod "d45ce109-ed30-459e-954f-452d1ab9e7c4" (UID: "d45ce109-ed30-459e-954f-452d1ab9e7c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:51.802879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.802853 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a255089b-e277-48a1-90b0-043db61a9443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98daec9f-hfcn7\" (UID: \"a255089b-e277-48a1-90b0-043db61a9443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7" Oct 30 22:09:51.803152 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.803127 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ljpq\" (UniqueName: \"kubernetes.io/projected/a255089b-e277-48a1-90b0-043db61a9443-kube-api-access-8ljpq\") pod \"maas-api-revoke-sub-98daec9f-hfcn7\" (UID: \"a255089b-e277-48a1-90b0-043db61a9443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7" Oct 30 22:09:51.803282 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.803262 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45ce109-ed30-459e-954f-452d1ab9e7c4-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:51.803952 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.803930 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d45ce109-ed30-459e-954f-452d1ab9e7c4-maas-api-xks-ca" pod "d45ce109-ed30-459e-954f-452d1ab9e7c4" (UID: "d45ce109-ed30-459e-954f-452d1ab9e7c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:51.804105 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.804087 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45ce109-ed30-459e-954f-452d1ab9e7c4-kube-api-access-fnvbb" pod "d45ce109-ed30-459e-954f-452d1ab9e7c4" (UID: "d45ce109-ed30-459e-954f-452d1ab9e7c4"). InnerVolumeSpecName "kube-api-access-fnvbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:51.904797 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.904763 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ljpq\" (UniqueName: \"kubernetes.io/projected/a255089b-e277-48a1-90b0-043db61a9443-kube-api-access-8ljpq\") pod \"maas-api-revoke-sub-98daec9f-hfcn7\" (UID: \"a255089b-e277-48a1-90b0-043db61a9443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7" Oct 30 22:09:51.904965 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.904949 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a255089b-e277-48a1-90b0-043db61a9443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98daec9f-hfcn7\" (UID: \"a255089b-e277-48a1-90b0-043db61a9443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7" Oct 30 22:09:51.905037 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.905014 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a255089b-e277-48a1-90b0-043db61a9443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98daec9f-hfcn7\" (UID: \"a255089b-e277-48a1-90b0-043db61a9443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7" Oct 30 22:09:51.905220 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.905203 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnvbb\" (UniqueName: \"kubernetes.io/projected/d45ce109-ed30-459e-954f-452d1ab9e7c4-kube-api-access-fnvbb\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:51.905292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.905225 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45ce109-ed30-459e-954f-452d1ab9e7c4-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:51.905602 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.905580 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a255089b-e277-48a1-90b0-043db61a9443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98daec9f-hfcn7\" (UID: \"a255089b-e277-48a1-90b0-043db61a9443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7" Oct 30 22:09:51.907320 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.907303 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a255089b-e277-48a1-90b0-043db61a9443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98daec9f-hfcn7\" (UID: \"a255089b-e277-48a1-90b0-043db61a9443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7" Oct 30 22:09:51.965858 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:51.965835 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ljpq\" (UniqueName: \"kubernetes.io/projected/a255089b-e277-48a1-90b0-043db61a9443-kube-api-access-8ljpq\") pod \"maas-api-revoke-sub-98daec9f-hfcn7\" (UID: \"a255089b-e277-48a1-90b0-043db61a9443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7" Oct 30 22:09:52.014818 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:52.014758 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7" Oct 30 22:09:52.400651 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:09:52.400589 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda255089b_e277_48a1_90b0_043db61a9443.slice/crio-915578a620d02b6560c256db1ca56c732f4ea3f4ab1d9a50677a50b78869207e WatchSource:0}: Error finding container 915578a620d02b6560c256db1ca56c732f4ea3f4ab1d9a50677a50b78869207e: Status 404 returned error can't find the container with id 915578a620d02b6560c256db1ca56c732f4ea3f4ab1d9a50677a50b78869207e Oct 30 22:09:52.405164 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:52.405135 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7"] Oct 30 22:09:52.492274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:52.492236 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7" event={"ID":"a255089b-e277-48a1-90b0-043db61a9443","Type":"ContainerStarted","Data":"915578a620d02b6560c256db1ca56c732f4ea3f4ab1d9a50677a50b78869207e"} Oct 30 22:09:52.494045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:52.494015 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s" Oct 30 22:09:52.494045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:52.494030 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s" event={"ID":"d45ce109-ed30-459e-954f-452d1ab9e7c4","Type":"ContainerDied","Data":"6980f0322037785440e025e10eb47e9f8c7049e0e9b3690f7ec64700a3ff43db"} Oct 30 22:09:52.494186 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:52.494070 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6980f0322037785440e025e10eb47e9f8c7049e0e9b3690f7ec64700a3ff43db" Oct 30 22:09:53.505777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:53.505744 4335 generic.go:356] "Generic (PLEG): container finished" podID="a255089b-e277-48a1-90b0-043db61a9443" containerID="a49bee525deea056c69c768d3523176701c6a17e31d098dc1bf4996e8ff3de2e" exitCode=0 Oct 30 22:09:53.506173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:53.505799 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7" event={"ID":"a255089b-e277-48a1-90b0-043db61a9443","Type":"ContainerDied","Data":"a49bee525deea056c69c768d3523176701c6a17e31d098dc1bf4996e8ff3de2e"} Oct 30 22:09:54.667112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:54.667077 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7" Oct 30 22:09:54.850482 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:54.850455 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a255089b-e277-48a1-90b0-043db61a9443-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a255089b-e277-48a1-90b0-043db61a9443-maas-api-xks-ca\") pod \"a255089b-e277-48a1-90b0-043db61a9443\" (UID: \"a255089b-e277-48a1-90b0-043db61a9443\") " Oct 30 22:09:54.850668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:54.850577 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a255089b-e277-48a1-90b0-043db61a9443-kube-api-access-8ljpq\" (UniqueName: \"kubernetes.io/projected/a255089b-e277-48a1-90b0-043db61a9443-kube-api-access-8ljpq\") pod \"a255089b-e277-48a1-90b0-043db61a9443\" (UID: \"a255089b-e277-48a1-90b0-043db61a9443\") " Oct 30 22:09:54.850729 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:54.850704 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a255089b-e277-48a1-90b0-043db61a9443-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a255089b-e277-48a1-90b0-043db61a9443-maas-api-openshift-ca\") pod \"a255089b-e277-48a1-90b0-043db61a9443\" (UID: \"a255089b-e277-48a1-90b0-043db61a9443\") " Oct 30 22:09:54.851092 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:54.851065 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a255089b-e277-48a1-90b0-043db61a9443-maas-api-openshift-ca" pod "a255089b-e277-48a1-90b0-043db61a9443" (UID: "a255089b-e277-48a1-90b0-043db61a9443"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:54.851429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:54.851409 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a255089b-e277-48a1-90b0-043db61a9443-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:54.852879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:54.852848 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a255089b-e277-48a1-90b0-043db61a9443-maas-api-xks-ca" pod "a255089b-e277-48a1-90b0-043db61a9443" (UID: "a255089b-e277-48a1-90b0-043db61a9443"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:54.852879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:54.852848 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a255089b-e277-48a1-90b0-043db61a9443-kube-api-access-8ljpq" pod "a255089b-e277-48a1-90b0-043db61a9443" (UID: "a255089b-e277-48a1-90b0-043db61a9443"). InnerVolumeSpecName "kube-api-access-8ljpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:54.952699 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:54.952667 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a255089b-e277-48a1-90b0-043db61a9443-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:54.952699 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:54.952701 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ljpq\" (UniqueName: \"kubernetes.io/projected/a255089b-e277-48a1-90b0-043db61a9443-kube-api-access-8ljpq\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:55.324684 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:09:55.324651 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:09:55.524463 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:55.524428 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7" event={"ID":"a255089b-e277-48a1-90b0-043db61a9443","Type":"ContainerDied","Data":"915578a620d02b6560c256db1ca56c732f4ea3f4ab1d9a50677a50b78869207e"} Oct 30 22:09:55.524645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:55.524476 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7" Oct 30 22:09:55.524645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:55.524484 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="915578a620d02b6560c256db1ca56c732f4ea3f4ab1d9a50677a50b78869207e" Oct 30 22:09:56.051640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.051576 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p"] Oct 30 22:09:56.056817 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.056795 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f274bbf6-7xk2p"] Oct 30 22:09:56.303105 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.303032 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2"] Oct 30 22:09:56.303915 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.303893 4335 cpu_manager.go:414] "containerMap: removing container" podUID="a255089b-e277-48a1-90b0-043db61a9443" containerName="revoke-keys" Oct 30 22:09:56.303915 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.303914 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a255089b-e277-48a1-90b0-043db61a9443" containerName="revoke-keys" Oct 30 22:09:56.304066 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.304053 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="a255089b-e277-48a1-90b0-043db61a9443" containerName="revoke-keys" Oct 30 22:09:56.308281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.308260 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.376560 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.376527 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2"] Oct 30 22:09:56.460756 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.460720 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv"] Oct 30 22:09:56.465094 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.465075 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv" Oct 30 22:09:56.472735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.472712 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f88e61-2539-422d-b204-cf54cec014f3-istiod-ca-cert\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.472862 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.472847 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-workload-certs\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.472932 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.472899 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwcjp\" (UniqueName: \"kubernetes.io/projected/88f88e61-2539-422d-b204-cf54cec014f3-kube-api-access-nwcjp\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.473000 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.472957 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f88e61-2539-422d-b204-cf54cec014f3-istio-token\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.473061 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.473009 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f88e61-2539-422d-b204-cf54cec014f3-istio-podinfo\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.473123 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.473072 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-istio-envoy\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.473273 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.473252 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-workload-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.473380 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.473294 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-credential-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.473380 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.473318 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-istio-data\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.517457 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.517211 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:56.517457 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.517296 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:56.517457 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.517396 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:09:56.517457 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.517427 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:56.523109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.523084 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv"] Oct 30 22:09:56.574473 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.574407 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d36705-49a0-44ac-b0fe-11f8bd39d06c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a82af8e-jkhkv\" (UID: \"24d36705-49a0-44ac-b0fe-11f8bd39d06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv" Oct 30 22:09:56.574588 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.574473 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwcjp\" (UniqueName: \"kubernetes.io/projected/88f88e61-2539-422d-b204-cf54cec014f3-kube-api-access-nwcjp\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.574588 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.574566 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f88e61-2539-422d-b204-cf54cec014f3-istio-token\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.574858 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.574840 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f88e61-2539-422d-b204-cf54cec014f3-istio-podinfo\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.574894 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.574866 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-istio-envoy\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.574933 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.574914 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-workload-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.574967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.574958 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d36705-49a0-44ac-b0fe-11f8bd39d06c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a82af8e-jkhkv\" (UID: \"24d36705-49a0-44ac-b0fe-11f8bd39d06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv" Oct 30 22:09:56.575013 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.574985 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-credential-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.575373 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.575349 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-workload-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.575373 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.575368 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-credential-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.575545 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.575399 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-istio-data\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.575545 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.575457 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f88e61-2539-422d-b204-cf54cec014f3-istiod-ca-cert\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.575545 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.575457 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-istio-data\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.575741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.575549 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4llns\" (UniqueName: \"kubernetes.io/projected/24d36705-49a0-44ac-b0fe-11f8bd39d06c-kube-api-access-4llns\") pod \"maas-api-revoke-sub-7a82af8e-jkhkv\" (UID: \"24d36705-49a0-44ac-b0fe-11f8bd39d06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv" Oct 30 22:09:56.575741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.575598 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-workload-certs\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.575877 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.575854 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-workload-certs\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.576346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.576323 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f88e61-2539-422d-b204-cf54cec014f3-istiod-ca-cert\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.577316 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.577297 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-istio-envoy\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.577554 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.577536 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f88e61-2539-422d-b204-cf54cec014f3-istio-podinfo\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.635526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.635504 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwcjp\" (UniqueName: \"kubernetes.io/projected/88f88e61-2539-422d-b204-cf54cec014f3-kube-api-access-nwcjp\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.635642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.635582 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f88e61-2539-422d-b204-cf54cec014f3-istio-token\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:56.676456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.676429 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4llns\" (UniqueName: \"kubernetes.io/projected/24d36705-49a0-44ac-b0fe-11f8bd39d06c-kube-api-access-4llns\") pod \"maas-api-revoke-sub-7a82af8e-jkhkv\" (UID: \"24d36705-49a0-44ac-b0fe-11f8bd39d06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv" Oct 30 22:09:56.676570 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.676497 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d36705-49a0-44ac-b0fe-11f8bd39d06c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a82af8e-jkhkv\" (UID: \"24d36705-49a0-44ac-b0fe-11f8bd39d06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv" Oct 30 22:09:56.676733 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.676670 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d36705-49a0-44ac-b0fe-11f8bd39d06c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a82af8e-jkhkv\" (UID: \"24d36705-49a0-44ac-b0fe-11f8bd39d06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv" Oct 30 22:09:56.677256 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.677234 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d36705-49a0-44ac-b0fe-11f8bd39d06c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a82af8e-jkhkv\" (UID: \"24d36705-49a0-44ac-b0fe-11f8bd39d06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv" Oct 30 22:09:56.678757 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.678740 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d36705-49a0-44ac-b0fe-11f8bd39d06c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a82af8e-jkhkv\" (UID: \"24d36705-49a0-44ac-b0fe-11f8bd39d06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv" Oct 30 22:09:56.737163 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.737133 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4llns\" (UniqueName: \"kubernetes.io/projected/24d36705-49a0-44ac-b0fe-11f8bd39d06c-kube-api-access-4llns\") pod \"maas-api-revoke-sub-7a82af8e-jkhkv\" (UID: \"24d36705-49a0-44ac-b0fe-11f8bd39d06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv" Oct 30 22:09:56.780001 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.779977 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv" Oct 30 22:09:56.925805 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:56.925766 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:57.060295 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:57.060262 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27824a71-6977-42de-afbb-c3578649bf70" path="/var/lib/kubelet/pods/27824a71-6977-42de-afbb-c3578649bf70/volumes" Oct 30 22:09:57.162946 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:09:57.162920 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d36705_49a0_44ac_b0fe_11f8bd39d06c.slice/crio-269be41c27c6b759a04830524022a37e06b928f79ac01550b205ae8ac251e884 WatchSource:0}: Error finding container 269be41c27c6b759a04830524022a37e06b928f79ac01550b205ae8ac251e884: Status 404 returned error can't find the container with id 269be41c27c6b759a04830524022a37e06b928f79ac01550b205ae8ac251e884 Oct 30 22:09:57.164019 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:57.163991 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv"] Oct 30 22:09:57.318040 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:09:57.318008 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f88e61_2539_422d_b204_cf54cec014f3.slice/crio-846c918482bc08e6d8f7162a5edd85480b0fbbb43229ab1f01ba1190a7603acc WatchSource:0}: Error finding container 846c918482bc08e6d8f7162a5edd85480b0fbbb43229ab1f01ba1190a7603acc: Status 404 returned error can't find the container with id 846c918482bc08e6d8f7162a5edd85480b0fbbb43229ab1f01ba1190a7603acc Oct 30 22:09:57.321232 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:57.321185 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:09:57.321330 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:57.321280 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:09:57.321370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:57.321325 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:09:57.322809 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:57.322783 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2"] Oct 30 22:09:57.548796 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:57.548764 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" event={"ID":"88f88e61-2539-422d-b204-cf54cec014f3","Type":"ContainerStarted","Data":"b4c977a14769edffa5b840b1b35a81ecdc750bbfc3c7c65deb1f28b97f134d66"} Oct 30 22:09:57.548935 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:57.548817 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" event={"ID":"88f88e61-2539-422d-b204-cf54cec014f3","Type":"ContainerStarted","Data":"846c918482bc08e6d8f7162a5edd85480b0fbbb43229ab1f01ba1190a7603acc"} Oct 30 22:09:57.550925 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:57.550900 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv" event={"ID":"24d36705-49a0-44ac-b0fe-11f8bd39d06c","Type":"ContainerStarted","Data":"269be41c27c6b759a04830524022a37e06b928f79ac01550b205ae8ac251e884"} Oct 30 22:09:57.671552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:57.671492 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" podStartSLOduration=1.671478236 podStartE2EDuration="1.671478236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:57.669409337 +0000 UTC m=+1530.769841068" watchObservedRunningTime="2026-10-30 22:09:57.671478236 +0000 UTC m=+1530.771909968" Oct 30 22:09:57.926181 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:57.926146 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:09:58.563446 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:58.563412 4335 generic.go:356] "Generic (PLEG): container finished" podID="24d36705-49a0-44ac-b0fe-11f8bd39d06c" containerID="fc0070fcdc0ab30b5518df27987e35ee1b7e7b7891a4595f65d7393fd52e88a0" exitCode=0 Oct 30 22:09:58.563892 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:58.563496 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv" event={"ID":"24d36705-49a0-44ac-b0fe-11f8bd39d06c","Type":"ContainerDied","Data":"fc0070fcdc0ab30b5518df27987e35ee1b7e7b7891a4595f65d7393fd52e88a0"} Oct 30 22:09:58.817690 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:58.802571 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv"] Oct 30 22:09:58.817690 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:58.817005 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv" Oct 30 22:09:58.862634 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:58.862594 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv"] Oct 30 22:09:58.938969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:58.938928 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:09:58.939179 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:58.938988 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" podUID="88f88e61-2539-422d-b204-cf54cec014f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 22:09:58.939907 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:58.939879 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d77bba1-2b42-4033-81af-e7d8a011bba7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c3d04b9-z6qtv\" (UID: \"2d77bba1-2b42-4033-81af-e7d8a011bba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv" Oct 30 22:09:58.940878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:58.940037 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d77bba1-2b42-4033-81af-e7d8a011bba7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c3d04b9-z6qtv\" (UID: \"2d77bba1-2b42-4033-81af-e7d8a011bba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv" Oct 30 22:09:58.940878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:58.940079 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c8sx\" (UniqueName: \"kubernetes.io/projected/2d77bba1-2b42-4033-81af-e7d8a011bba7-kube-api-access-2c8sx\") pod \"maas-api-revoke-sub-0c3d04b9-z6qtv\" (UID: \"2d77bba1-2b42-4033-81af-e7d8a011bba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv" Oct 30 22:09:59.044059 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.044019 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d77bba1-2b42-4033-81af-e7d8a011bba7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c3d04b9-z6qtv\" (UID: \"2d77bba1-2b42-4033-81af-e7d8a011bba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv" Oct 30 22:09:59.044553 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.044520 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d77bba1-2b42-4033-81af-e7d8a011bba7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c3d04b9-z6qtv\" (UID: \"2d77bba1-2b42-4033-81af-e7d8a011bba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv" Oct 30 22:09:59.044711 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.044587 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8sx\" (UniqueName: \"kubernetes.io/projected/2d77bba1-2b42-4033-81af-e7d8a011bba7-kube-api-access-2c8sx\") pod \"maas-api-revoke-sub-0c3d04b9-z6qtv\" (UID: \"2d77bba1-2b42-4033-81af-e7d8a011bba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv" Oct 30 22:09:59.046576 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.046543 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d77bba1-2b42-4033-81af-e7d8a011bba7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c3d04b9-z6qtv\" (UID: \"2d77bba1-2b42-4033-81af-e7d8a011bba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv" Oct 30 22:09:59.053069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.053041 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d77bba1-2b42-4033-81af-e7d8a011bba7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c3d04b9-z6qtv\" (UID: \"2d77bba1-2b42-4033-81af-e7d8a011bba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv" Oct 30 22:09:59.130656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.130554 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c8sx\" (UniqueName: \"kubernetes.io/projected/2d77bba1-2b42-4033-81af-e7d8a011bba7-kube-api-access-2c8sx\") pod \"maas-api-revoke-sub-0c3d04b9-z6qtv\" (UID: \"2d77bba1-2b42-4033-81af-e7d8a011bba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv" Oct 30 22:09:59.140515 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.140493 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv" Oct 30 22:09:59.547558 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.547532 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv"] Oct 30 22:09:59.549732 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:09:59.549706 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d77bba1_2b42_4033_81af_e7d8a011bba7.slice/crio-6d98cd022b4b1aaf13b6946821b1ab993c4cddc53ab751214b7291525cc5d7ab WatchSource:0}: Error finding container 6d98cd022b4b1aaf13b6946821b1ab993c4cddc53ab751214b7291525cc5d7ab: Status 404 returned error can't find the container with id 6d98cd022b4b1aaf13b6946821b1ab993c4cddc53ab751214b7291525cc5d7ab Oct 30 22:09:59.575470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.575442 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv" event={"ID":"2d77bba1-2b42-4033-81af-e7d8a011bba7","Type":"ContainerStarted","Data":"6d98cd022b4b1aaf13b6946821b1ab993c4cddc53ab751214b7291525cc5d7ab"} Oct 30 22:09:59.766733 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.766701 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv" Oct 30 22:09:59.866266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.866241 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24d36705-49a0-44ac-b0fe-11f8bd39d06c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d36705-49a0-44ac-b0fe-11f8bd39d06c-maas-api-openshift-ca\") pod \"24d36705-49a0-44ac-b0fe-11f8bd39d06c\" (UID: \"24d36705-49a0-44ac-b0fe-11f8bd39d06c\") " Oct 30 22:09:59.866432 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.866421 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d36705-49a0-44ac-b0fe-11f8bd39d06c-kube-api-access-4llns\" (UniqueName: \"kubernetes.io/projected/24d36705-49a0-44ac-b0fe-11f8bd39d06c-kube-api-access-4llns\") pod \"24d36705-49a0-44ac-b0fe-11f8bd39d06c\" (UID: \"24d36705-49a0-44ac-b0fe-11f8bd39d06c\") " Oct 30 22:09:59.866479 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.866448 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24d36705-49a0-44ac-b0fe-11f8bd39d06c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d36705-49a0-44ac-b0fe-11f8bd39d06c-maas-api-xks-ca\") pod \"24d36705-49a0-44ac-b0fe-11f8bd39d06c\" (UID: \"24d36705-49a0-44ac-b0fe-11f8bd39d06c\") " Oct 30 22:09:59.866584 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.866552 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d36705-49a0-44ac-b0fe-11f8bd39d06c-maas-api-openshift-ca" pod "24d36705-49a0-44ac-b0fe-11f8bd39d06c" (UID: "24d36705-49a0-44ac-b0fe-11f8bd39d06c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:59.866813 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.866796 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d36705-49a0-44ac-b0fe-11f8bd39d06c-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:59.868652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.868609 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d36705-49a0-44ac-b0fe-11f8bd39d06c-maas-api-xks-ca" pod "24d36705-49a0-44ac-b0fe-11f8bd39d06c" (UID: "24d36705-49a0-44ac-b0fe-11f8bd39d06c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:59.868740 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.868665 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d36705-49a0-44ac-b0fe-11f8bd39d06c-kube-api-access-4llns" pod "24d36705-49a0-44ac-b0fe-11f8bd39d06c" (UID: "24d36705-49a0-44ac-b0fe-11f8bd39d06c"). InnerVolumeSpecName "kube-api-access-4llns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:59.933644 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.933260 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:09:59.933644 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.933318 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" podUID="88f88e61-2539-422d-b204-cf54cec014f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 22:09:59.968138 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.968105 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4llns\" (UniqueName: \"kubernetes.io/projected/24d36705-49a0-44ac-b0fe-11f8bd39d06c-kube-api-access-4llns\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:59.968138 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:09:59.968141 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d36705-49a0-44ac-b0fe-11f8bd39d06c-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:00.586100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:00.586066 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv" Oct 30 22:10:00.586100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:00.586094 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv" event={"ID":"24d36705-49a0-44ac-b0fe-11f8bd39d06c","Type":"ContainerDied","Data":"269be41c27c6b759a04830524022a37e06b928f79ac01550b205ae8ac251e884"} Oct 30 22:10:00.586594 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:00.586140 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="269be41c27c6b759a04830524022a37e06b928f79ac01550b205ae8ac251e884" Oct 30 22:10:00.588273 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:00.588246 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv" event={"ID":"2d77bba1-2b42-4033-81af-e7d8a011bba7","Type":"ContainerStarted","Data":"22bbc2228ae9223eaa5a1fc913593d026d49e8cc20b3f75170049ca320e5c7d0"} Oct 30 22:10:00.728843 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:00.726589 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv" podStartSLOduration=2.726569644 podStartE2EDuration="2.726569644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:00.711049362 +0000 UTC m=+1533.811481095" watchObservedRunningTime="2026-10-30 22:10:00.726569644 +0000 UTC m=+1533.827001377" Oct 30 22:10:00.926997 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:00.926904 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:10:00.926997 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:00.926968 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" podUID="88f88e61-2539-422d-b204-cf54cec014f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 22:10:01.628521 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:01.619573 4335 generic.go:356] "Generic (PLEG): container finished" podID="2d77bba1-2b42-4033-81af-e7d8a011bba7" containerID="22bbc2228ae9223eaa5a1fc913593d026d49e8cc20b3f75170049ca320e5c7d0" exitCode=0 Oct 30 22:10:01.628521 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:01.619852 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv" event={"ID":"2d77bba1-2b42-4033-81af-e7d8a011bba7","Type":"ContainerDied","Data":"22bbc2228ae9223eaa5a1fc913593d026d49e8cc20b3f75170049ca320e5c7d0"} Oct 30 22:10:01.926800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:01.926717 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:10:01.926800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:01.926771 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" podUID="88f88e61-2539-422d-b204-cf54cec014f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 22:10:02.884538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:02.884509 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv" Oct 30 22:10:02.945448 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:02.945338 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:10:02.945448 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:02.945419 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" podUID="88f88e61-2539-422d-b204-cf54cec014f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 22:10:02.949126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:02.949097 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d77bba1-2b42-4033-81af-e7d8a011bba7-kube-api-access-2c8sx\" (UniqueName: \"kubernetes.io/projected/2d77bba1-2b42-4033-81af-e7d8a011bba7-kube-api-access-2c8sx\") pod \"2d77bba1-2b42-4033-81af-e7d8a011bba7\" (UID: \"2d77bba1-2b42-4033-81af-e7d8a011bba7\") " Oct 30 22:10:02.949414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:02.949393 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d77bba1-2b42-4033-81af-e7d8a011bba7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d77bba1-2b42-4033-81af-e7d8a011bba7-maas-api-openshift-ca\") pod \"2d77bba1-2b42-4033-81af-e7d8a011bba7\" (UID: \"2d77bba1-2b42-4033-81af-e7d8a011bba7\") " Oct 30 22:10:02.949497 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:02.949438 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d77bba1-2b42-4033-81af-e7d8a011bba7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d77bba1-2b42-4033-81af-e7d8a011bba7-maas-api-xks-ca\") pod \"2d77bba1-2b42-4033-81af-e7d8a011bba7\" (UID: \"2d77bba1-2b42-4033-81af-e7d8a011bba7\") " Oct 30 22:10:02.950929 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:02.950894 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4"] Oct 30 22:10:02.952055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:02.952015 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d77bba1-2b42-4033-81af-e7d8a011bba7-maas-api-openshift-ca" pod "2d77bba1-2b42-4033-81af-e7d8a011bba7" (UID: "2d77bba1-2b42-4033-81af-e7d8a011bba7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:02.953177 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:02.953158 4335 cpu_manager.go:414] "containerMap: removing container" podUID="24d36705-49a0-44ac-b0fe-11f8bd39d06c" containerName="revoke-keys" Oct 30 22:10:02.953298 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:02.953285 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d36705-49a0-44ac-b0fe-11f8bd39d06c" containerName="revoke-keys" Oct 30 22:10:02.953408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:02.953396 4335 cpu_manager.go:414] "containerMap: removing container" podUID="2d77bba1-2b42-4033-81af-e7d8a011bba7" containerName="revoke-keys" Oct 30 22:10:02.954334 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:02.954316 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d77bba1-2b42-4033-81af-e7d8a011bba7" containerName="revoke-keys" Oct 30 22:10:02.954731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:02.954714 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d36705-49a0-44ac-b0fe-11f8bd39d06c" containerName="revoke-keys" Oct 30 22:10:02.954845 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:02.954835 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d77bba1-2b42-4033-81af-e7d8a011bba7" containerName="revoke-keys" Oct 30 22:10:02.958988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:02.958970 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d77bba1-2b42-4033-81af-e7d8a011bba7-kube-api-access-2c8sx" pod "2d77bba1-2b42-4033-81af-e7d8a011bba7" (UID: "2d77bba1-2b42-4033-81af-e7d8a011bba7"). InnerVolumeSpecName "kube-api-access-2c8sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:02.959715 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:02.959546 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d77bba1-2b42-4033-81af-e7d8a011bba7-maas-api-xks-ca" pod "2d77bba1-2b42-4033-81af-e7d8a011bba7" (UID: "2d77bba1-2b42-4033-81af-e7d8a011bba7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:02.959948 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:02.959911 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.080345 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.079066 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-workload-socket\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.080345 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.079154 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-credential-socket\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.080345 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.079209 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-envoy\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.080345 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.079359 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-token\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.080345 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.079483 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-data\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.080345 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.079546 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-workload-certs\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.080345 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.079588 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfqts\" (UniqueName: \"kubernetes.io/projected/ef6c24d9-0e05-4181-9c98-dd5015f58975-kube-api-access-bfqts\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.080345 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.079785 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-podinfo\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.080345 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.079837 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6c24d9-0e05-4181-9c98-dd5015f58975-istiod-ca-cert\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.081684 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.081660 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c8sx\" (UniqueName: \"kubernetes.io/projected/2d77bba1-2b42-4033-81af-e7d8a011bba7-kube-api-access-2c8sx\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:03.092937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.087283 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d77bba1-2b42-4033-81af-e7d8a011bba7-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:03.092937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.087327 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d77bba1-2b42-4033-81af-e7d8a011bba7-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:03.152884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.150077 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4"] Oct 30 22:10:03.162955 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.162906 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88"] Oct 30 22:10:03.171386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.171357 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb66c93f-plx88"] Oct 30 22:10:03.189482 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.189455 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-token\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.189589 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.189557 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-data\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.189718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.189604 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-workload-certs\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.189718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.189670 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfqts\" (UniqueName: \"kubernetes.io/projected/ef6c24d9-0e05-4181-9c98-dd5015f58975-kube-api-access-bfqts\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.189841 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.189810 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-podinfo\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.189899 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.189842 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6c24d9-0e05-4181-9c98-dd5015f58975-istiod-ca-cert\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.189961 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.189913 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-workload-socket\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.190026 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.189963 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-credential-socket\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.190026 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.190002 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-envoy\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.190224 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.190186 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-workload-certs\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.190326 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.190302 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-workload-socket\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.190521 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.190501 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-credential-socket\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.190733 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.190713 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-data\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.190960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.190942 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6c24d9-0e05-4181-9c98-dd5015f58975-istiod-ca-cert\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.193070 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.193033 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-envoy\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.193381 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.193357 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-podinfo\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.251001 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.250933 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-token\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.251333 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.251313 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfqts\" (UniqueName: \"kubernetes.io/projected/ef6c24d9-0e05-4181-9c98-dd5015f58975-kube-api-access-bfqts\") pod \"e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.454909 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.454881 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:03.663524 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.663487 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv" event={"ID":"2d77bba1-2b42-4033-81af-e7d8a011bba7","Type":"ContainerDied","Data":"6d98cd022b4b1aaf13b6946821b1ab993c4cddc53ab751214b7291525cc5d7ab"} Oct 30 22:10:03.663735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.663549 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d98cd022b4b1aaf13b6946821b1ab993c4cddc53ab751214b7291525cc5d7ab" Oct 30 22:10:03.663735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.663500 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv" Oct 30 22:10:03.926775 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.926746 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:10:03.930931 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.926820 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" podUID="88f88e61-2539-422d-b204-cf54cec014f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 22:10:03.930931 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:10:03.929228 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6c24d9_0e05_4181_9c98_dd5015f58975.slice/crio-e28ce7add067438abdfab973f3711c36b0c080280e15b085177c179d7863c9a4 WatchSource:0}: Error finding container e28ce7add067438abdfab973f3711c36b0c080280e15b085177c179d7863c9a4: Status 404 returned error can't find the container with id e28ce7add067438abdfab973f3711c36b0c080280e15b085177c179d7863c9a4 Oct 30 22:10:03.930931 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.930268 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4"] Oct 30 22:10:03.932859 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.932821 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:10:03.932980 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.932908 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:10:03.932980 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:03.932954 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:10:04.676188 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:04.676150 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" event={"ID":"ef6c24d9-0e05-4181-9c98-dd5015f58975","Type":"ContainerStarted","Data":"e38e6d94fa6972341299b9a7913f8f299846ebc5d735f19feb304ce438a7db38"} Oct 30 22:10:04.676376 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:04.676215 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" event={"ID":"ef6c24d9-0e05-4181-9c98-dd5015f58975","Type":"ContainerStarted","Data":"e28ce7add067438abdfab973f3711c36b0c080280e15b085177c179d7863c9a4"} Oct 30 22:10:04.798471 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:04.798420 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" podStartSLOduration=2.798401601 podStartE2EDuration="2.798401601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:04.796326201 +0000 UTC m=+1537.896757984" watchObservedRunningTime="2026-10-30 22:10:04.798401601 +0000 UTC m=+1537.898833332" Oct 30 22:10:05.065795 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:05.065754 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7175d8a3-d4af-4f8c-87eb-35c096f00e77" path="/var/lib/kubelet/pods/7175d8a3-d4af-4f8c-87eb-35c096f00e77/volumes" Oct 30 22:10:05.179759 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:05.179728 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:10:05.180361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:05.180339 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:10:05.181280 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:05.181261 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:10:05.455696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:05.455459 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:05.456840 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:05.456809 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:10:05.456959 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:05.456878 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" podUID="ef6c24d9-0e05-4181-9c98-dd5015f58975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 22:10:05.733510 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:05.733420 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6954-bc401f"] Oct 30 22:10:05.738813 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:05.737689 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6954-bc401f" Oct 30 22:10:05.789461 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:05.789434 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6954-bc401f"] Oct 30 22:10:05.817115 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:05.817085 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-z48h6\"" Oct 30 22:10:05.843038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:05.842999 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cfgp\" (UniqueName: \"kubernetes.io/projected/7fd7f856-45c5-4a3c-8622-c15c66fb7f90-kube-api-access-7cfgp\") pod \"test-curl-6954-bc401f\" (UID: \"7fd7f856-45c5-4a3c-8622-c15c66fb7f90\") " pod="opendatahub/test-curl-6954-bc401f" Oct 30 22:10:05.944155 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:05.944113 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cfgp\" (UniqueName: \"kubernetes.io/projected/7fd7f856-45c5-4a3c-8622-c15c66fb7f90-kube-api-access-7cfgp\") pod \"test-curl-6954-bc401f\" (UID: \"7fd7f856-45c5-4a3c-8622-c15c66fb7f90\") " pod="opendatahub/test-curl-6954-bc401f" Oct 30 22:10:06.028711 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:06.019313 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cfgp\" (UniqueName: \"kubernetes.io/projected/7fd7f856-45c5-4a3c-8622-c15c66fb7f90-kube-api-access-7cfgp\") pod \"test-curl-6954-bc401f\" (UID: \"7fd7f856-45c5-4a3c-8622-c15c66fb7f90\") " pod="opendatahub/test-curl-6954-bc401f" Oct 30 22:10:06.053093 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:06.053067 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6954-bc401f" Oct 30 22:10:06.455584 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:06.455554 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:10:06.456023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:06.455609 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" podUID="ef6c24d9-0e05-4181-9c98-dd5015f58975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 22:10:06.465732 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:10:06.465691 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fd7f856_45c5_4a3c_8622_c15c66fb7f90.slice/crio-41cbe95180b29c0bc6438e08ef4db584dd3edb3b2be62a0c7256a98e092aa9ab WatchSource:0}: Error finding container 41cbe95180b29c0bc6438e08ef4db584dd3edb3b2be62a0c7256a98e092aa9ab: Status 404 returned error can't find the container with id 41cbe95180b29c0bc6438e08ef4db584dd3edb3b2be62a0c7256a98e092aa9ab Oct 30 22:10:06.468147 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:06.468121 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6954-bc401f"] Oct 30 22:10:06.468318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:06.468302 4335 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:10:06.726099 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:06.726020 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6954-bc401f" event={"ID":"7fd7f856-45c5-4a3c-8622-c15c66fb7f90","Type":"ContainerStarted","Data":"41cbe95180b29c0bc6438e08ef4db584dd3edb3b2be62a0c7256a98e092aa9ab"} Oct 30 22:10:07.462642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:07.461860 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:10:07.462642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:07.461946 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" podUID="ef6c24d9-0e05-4181-9c98-dd5015f58975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 22:10:08.456642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:08.456071 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:10:08.456642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:08.456132 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" podUID="ef6c24d9-0e05-4181-9c98-dd5015f58975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 22:10:08.726707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:08.724750 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:08.827673 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:08.818794 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:09.041689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.028429 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:09.258966 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.256035 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:09.258966 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.258075 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:09.371704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.367431 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:09.413667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.382361 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:09.413667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.382828 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Oct 30 22:10:09.413667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.383152 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": context deadline exceeded" Oct 30 22:10:09.413667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.383952 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:09.413667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.410761 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:09.417649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.415130 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:09.479661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.460581 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 30 22:10:09.479661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.461728 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:10:09.479661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.461772 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" podUID="ef6c24d9-0e05-4181-9c98-dd5015f58975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 22:10:09.537748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.537700 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:09.757695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.729944 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 30 22:10:09.757695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.730401 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:10:09.758381 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.758337 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:10.014715 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.986427 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Oct 30 22:10:10.014715 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:09.987843 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:10:10.081300 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.081254 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:10.123723 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.123691 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:10:10.370647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.369418 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:10.370647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.370474 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:10:10.410538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.384553 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:10.410538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.384956 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:10:10.410538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.385868 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:10.410538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.390779 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:10.410538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.390999 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Oct 30 22:10:10.410538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.391192 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:10:10.410538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.391245 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:10:10.410538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.391280 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:10:10.429119 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.429082 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:10.455643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.430254 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 22:10:10.455643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.446983 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 22:10:10.455643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.447288 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:10:10.481638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.479575 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:10:10.481638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.479656 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" podUID="ef6c24d9-0e05-4181-9c98-dd5015f58975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 22:10:10.481638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.480162 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 30 22:10:10.481638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.480490 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:10:10.746817 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.746709 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:10.818685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.769961 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:10.827643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:10.824032 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:10:11.012780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:11.011696 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:11.099840 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:11.099798 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:11.173509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:11.173471 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 22:10:11.334053 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:11.334014 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/health\": context deadline exceeded" Oct 30 22:10:11.377685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:11.376758 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:11.424656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:11.396825 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Oct 30 22:10:11.424656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:11.397185 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Oct 30 22:10:11.424656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:11.397434 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": context deadline exceeded" Oct 30 22:10:11.424656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:11.397685 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 22:10:11.429968 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:11.429928 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:11.460771 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:11.460702 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:11.465652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:11.461692 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:10:11.465652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:11.461738 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" podUID="ef6c24d9-0e05-4181-9c98-dd5015f58975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 22:10:11.521105 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:11.508730 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 30 22:10:11.870643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:11.839724 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 30 22:10:11.870643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:11.841738 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:12.122699 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:12.091648 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Oct 30 22:10:12.180783 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:12.180429 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:12.441695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:12.435007 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:12.457751 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:12.457701 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:12.499642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:12.459953 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:12.525649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:12.461710 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:12.525649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:12.461949 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:12.525649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:12.461979 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Oct 30 22:10:12.525649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:12.462009 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:12.525649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:12.462313 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:10:12.525649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:12.512046 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" podUID="ef6c24d9-0e05-4181-9c98-dd5015f58975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 22:10:12.525649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:12.512917 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:12.858428 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:12.858388 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:12.858739 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:12.858699 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 30 22:10:12.996782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:12.996725 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:13.259700 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.212857 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Oct 30 22:10:13.259700 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.214860 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:13.307206 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.307145 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:10:13.310787 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.310746 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" podUID="04c45673-5196-4efe-a65c-fbf6ba5e3c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:13.440785 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.440743 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 22:10:13.487305 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.463279 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:10:13.487305 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.463349 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" podUID="ef6c24d9-0e05-4181-9c98-dd5015f58975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 22:10:13.505173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.503714 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:13.505173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.504166 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:13.505173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.504454 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:13.535767 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.532867 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:13.535767 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.534107 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-lhszx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:10:13.535767 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.534160 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:13.535767 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.534249 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:13.535767 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.534541 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Oct 30 22:10:13.537033 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.536987 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": context deadline exceeded" Oct 30 22:10:13.595968 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.595929 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-6bt25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:10:13.596917 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.596860 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:13.822642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.818651 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:13.860288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.860248 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 30 22:10:13.860676 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.860550 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:13.907327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.891758 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:10:13.907327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:13.891824 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" podUID="87b2369a-d31f-4413-b5c6-ec4b217c3f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:14.214522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:14.214247 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Oct 30 22:10:14.215860 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:14.215797 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:14.680640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:14.649151 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Oct 30 22:10:14.680640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:14.649543 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:14.691643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:14.681860 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:14.691643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:14.682175 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 22:10:14.691643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:14.682422 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:14.711919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:14.692329 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:10:14.711919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:14.692376 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" podUID="ef6c24d9-0e05-4181-9c98-dd5015f58975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 22:10:14.711919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:14.694302 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": context deadline exceeded" Oct 30 22:10:14.711919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:14.694571 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Oct 30 22:10:14.711919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:14.706203 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 22:10:14.713817 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:14.713771 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:10:14.714094 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:14.714072 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" podUID="5c5448fc-7500-43b9-abd3-f9a8c603fbd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:14.867972 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:14.867918 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:14.882037 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:14.881999 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:15.043277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:15.043177 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:15.229640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:15.218014 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:15.229640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:15.218393 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:15.385856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:15.385830 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:10:15.459840 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:15.459699 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:10:15.459840 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:15.459762 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" podUID="ef6c24d9-0e05-4181-9c98-dd5015f58975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 22:10:15.578582 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:15.578425 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:10:15.589176 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:15.588503 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:10:15.598664 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:15.598289 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:10:15.638528 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:15.636048 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:10:15.654686 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:15.652608 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:15.662656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:15.658902 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:10:15.662656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:15.660143 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 22:10:15.662656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:15.661278 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:10:15.666910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:15.666890 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:10:15.667747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:15.667729 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:10:15.681627 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:15.676055 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:10:15.707042 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:15.706926 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:10:16.456885 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:16.456768 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:10:16.456885 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:16.456830 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" podUID="ef6c24d9-0e05-4181-9c98-dd5015f58975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 22:10:18.461829 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:18.461803 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:18.504227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:18.503790 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:10:18.504227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:18.503845 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" podUID="ef6c24d9-0e05-4181-9c98-dd5015f58975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:18.747988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:18.747915 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:19.436184 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.436145 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6954-bc401f" event={"ID":"7fd7f856-45c5-4a3c-8622-c15c66fb7f90","Type":"ContainerStarted","Data":"df85b16372adf38100e3cb851afd50866637c8db3699225fd164d67263119659"} Oct 30 22:10:19.437371 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.437347 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:10:19.557955 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.557925 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh"] Oct 30 22:10:19.562946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.562929 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh" Oct 30 22:10:19.601356 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.601312 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6954-bc401f" podStartSLOduration=2.627996923 podStartE2EDuration="14.601301552s" totalImagesPullingTime="11.973304629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:19.556105942 +0000 UTC m=+1552.656537670" watchObservedRunningTime="2026-10-30 22:10:19.601301552 +0000 UTC m=+1552.701733283" Oct 30 22:10:19.620856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.620829 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh"] Oct 30 22:10:19.650772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.650744 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:19.657824 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.657801 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:10:19.658004 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.657964 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:10:19.658080 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.658042 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:10:19.699769 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.699693 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lhm9\" (UniqueName: \"kubernetes.io/projected/dcc083f7-11a5-476a-8124-5d38fc2b0468-kube-api-access-5lhm9\") pod \"maas-api-revoke-sub-4cbc2e92-ncjqh\" (UID: \"dcc083f7-11a5-476a-8124-5d38fc2b0468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh" Oct 30 22:10:19.699913 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.699810 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc083f7-11a5-476a-8124-5d38fc2b0468-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cbc2e92-ncjqh\" (UID: \"dcc083f7-11a5-476a-8124-5d38fc2b0468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh" Oct 30 22:10:19.699976 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.699930 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc083f7-11a5-476a-8124-5d38fc2b0468-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cbc2e92-ncjqh\" (UID: \"dcc083f7-11a5-476a-8124-5d38fc2b0468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh" Oct 30 22:10:19.801760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.801718 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc083f7-11a5-476a-8124-5d38fc2b0468-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cbc2e92-ncjqh\" (UID: \"dcc083f7-11a5-476a-8124-5d38fc2b0468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh" Oct 30 22:10:19.801968 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.801952 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lhm9\" (UniqueName: \"kubernetes.io/projected/dcc083f7-11a5-476a-8124-5d38fc2b0468-kube-api-access-5lhm9\") pod \"maas-api-revoke-sub-4cbc2e92-ncjqh\" (UID: \"dcc083f7-11a5-476a-8124-5d38fc2b0468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh" Oct 30 22:10:19.802068 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.802052 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc083f7-11a5-476a-8124-5d38fc2b0468-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cbc2e92-ncjqh\" (UID: \"dcc083f7-11a5-476a-8124-5d38fc2b0468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh" Oct 30 22:10:19.803316 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.803288 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc083f7-11a5-476a-8124-5d38fc2b0468-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cbc2e92-ncjqh\" (UID: \"dcc083f7-11a5-476a-8124-5d38fc2b0468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh" Oct 30 22:10:19.804777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.804748 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc083f7-11a5-476a-8124-5d38fc2b0468-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cbc2e92-ncjqh\" (UID: \"dcc083f7-11a5-476a-8124-5d38fc2b0468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh" Oct 30 22:10:19.876538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.876510 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lhm9\" (UniqueName: \"kubernetes.io/projected/dcc083f7-11a5-476a-8124-5d38fc2b0468-kube-api-access-5lhm9\") pod \"maas-api-revoke-sub-4cbc2e92-ncjqh\" (UID: \"dcc083f7-11a5-476a-8124-5d38fc2b0468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh" Oct 30 22:10:19.876797 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:19.876778 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh" Oct 30 22:10:20.284677 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:10:20.284605 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc083f7_11a5_476a_8124_5d38fc2b0468.slice/crio-dbfc674298ee423dbea89cf3d71e6cda7089060cef2118b20f570b594bbbe5db WatchSource:0}: Error finding container dbfc674298ee423dbea89cf3d71e6cda7089060cef2118b20f570b594bbbe5db: Status 404 returned error can't find the container with id dbfc674298ee423dbea89cf3d71e6cda7089060cef2118b20f570b594bbbe5db Oct 30 22:10:20.287415 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:20.287386 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh"] Oct 30 22:10:20.449245 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:20.449209 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh" event={"ID":"dcc083f7-11a5-476a-8124-5d38fc2b0468","Type":"ContainerStarted","Data":"dbfc674298ee423dbea89cf3d71e6cda7089060cef2118b20f570b594bbbe5db"} Oct 30 22:10:21.460629 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:21.460584 4335 generic.go:356] "Generic (PLEG): container finished" podID="dcc083f7-11a5-476a-8124-5d38fc2b0468" containerID="8b349a34e4d208080d13b79622da8e84dd519cbd8339606fbd258d738f63082c" exitCode=0 Oct 30 22:10:21.461017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:21.460645 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh" event={"ID":"dcc083f7-11a5-476a-8124-5d38fc2b0468","Type":"ContainerDied","Data":"8b349a34e4d208080d13b79622da8e84dd519cbd8339606fbd258d738f63082c"} Oct 30 22:10:22.680811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.680788 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh" Oct 30 22:10:22.792917 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.792853 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv"] Oct 30 22:10:22.794122 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.794096 4335 cpu_manager.go:414] "containerMap: removing container" podUID="dcc083f7-11a5-476a-8124-5d38fc2b0468" containerName="revoke-keys" Oct 30 22:10:22.794122 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.794122 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc083f7-11a5-476a-8124-5d38fc2b0468" containerName="revoke-keys" Oct 30 22:10:22.794346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.794331 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc083f7-11a5-476a-8124-5d38fc2b0468" containerName="revoke-keys" Oct 30 22:10:22.798644 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.798591 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv" Oct 30 22:10:22.844065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.844030 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcc083f7-11a5-476a-8124-5d38fc2b0468-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc083f7-11a5-476a-8124-5d38fc2b0468-maas-api-xks-ca\") pod \"dcc083f7-11a5-476a-8124-5d38fc2b0468\" (UID: \"dcc083f7-11a5-476a-8124-5d38fc2b0468\") " Oct 30 22:10:22.844198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.844178 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc083f7-11a5-476a-8124-5d38fc2b0468-kube-api-access-5lhm9\" (UniqueName: \"kubernetes.io/projected/dcc083f7-11a5-476a-8124-5d38fc2b0468-kube-api-access-5lhm9\") pod \"dcc083f7-11a5-476a-8124-5d38fc2b0468\" (UID: \"dcc083f7-11a5-476a-8124-5d38fc2b0468\") " Oct 30 22:10:22.844353 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.844332 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcc083f7-11a5-476a-8124-5d38fc2b0468-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc083f7-11a5-476a-8124-5d38fc2b0468-maas-api-openshift-ca\") pod \"dcc083f7-11a5-476a-8124-5d38fc2b0468\" (UID: \"dcc083f7-11a5-476a-8124-5d38fc2b0468\") " Oct 30 22:10:22.845494 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.845468 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcc083f7-11a5-476a-8124-5d38fc2b0468-maas-api-openshift-ca" pod "dcc083f7-11a5-476a-8124-5d38fc2b0468" (UID: "dcc083f7-11a5-476a-8124-5d38fc2b0468"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:22.876821 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.856723 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv"] Oct 30 22:10:22.876821 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.871749 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc083f7-11a5-476a-8124-5d38fc2b0468-kube-api-access-5lhm9" pod "dcc083f7-11a5-476a-8124-5d38fc2b0468" (UID: "dcc083f7-11a5-476a-8124-5d38fc2b0468"). InnerVolumeSpecName "kube-api-access-5lhm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:22.876821 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.871840 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcc083f7-11a5-476a-8124-5d38fc2b0468-maas-api-xks-ca" pod "dcc083f7-11a5-476a-8124-5d38fc2b0468" (UID: "dcc083f7-11a5-476a-8124-5d38fc2b0468"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:22.947191 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.947161 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c0902f1-2ad0-4787-986f-3519210cb168-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5702beb6-gcxfv\" (UID: \"7c0902f1-2ad0-4787-986f-3519210cb168\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv" Oct 30 22:10:22.947331 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.947219 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npxnl\" (UniqueName: \"kubernetes.io/projected/7c0902f1-2ad0-4787-986f-3519210cb168-kube-api-access-npxnl\") pod \"maas-api-revoke-sub-5702beb6-gcxfv\" (UID: \"7c0902f1-2ad0-4787-986f-3519210cb168\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv" Oct 30 22:10:22.947371 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.947354 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c0902f1-2ad0-4787-986f-3519210cb168-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5702beb6-gcxfv\" (UID: \"7c0902f1-2ad0-4787-986f-3519210cb168\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv" Oct 30 22:10:22.947572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.947553 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lhm9\" (UniqueName: \"kubernetes.io/projected/dcc083f7-11a5-476a-8124-5d38fc2b0468-kube-api-access-5lhm9\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:22.947629 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.947575 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc083f7-11a5-476a-8124-5d38fc2b0468-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:22.947629 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:22.947587 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc083f7-11a5-476a-8124-5d38fc2b0468-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:23.049130 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:23.049112 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c0902f1-2ad0-4787-986f-3519210cb168-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5702beb6-gcxfv\" (UID: \"7c0902f1-2ad0-4787-986f-3519210cb168\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv" Oct 30 22:10:23.049261 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:23.049148 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npxnl\" (UniqueName: \"kubernetes.io/projected/7c0902f1-2ad0-4787-986f-3519210cb168-kube-api-access-npxnl\") pod \"maas-api-revoke-sub-5702beb6-gcxfv\" (UID: \"7c0902f1-2ad0-4787-986f-3519210cb168\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv" Oct 30 22:10:23.049261 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:23.049186 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c0902f1-2ad0-4787-986f-3519210cb168-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5702beb6-gcxfv\" (UID: \"7c0902f1-2ad0-4787-986f-3519210cb168\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv" Oct 30 22:10:23.049894 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:23.049866 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c0902f1-2ad0-4787-986f-3519210cb168-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5702beb6-gcxfv\" (UID: \"7c0902f1-2ad0-4787-986f-3519210cb168\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv" Oct 30 22:10:23.051941 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:23.051914 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c0902f1-2ad0-4787-986f-3519210cb168-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5702beb6-gcxfv\" (UID: \"7c0902f1-2ad0-4787-986f-3519210cb168\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv" Oct 30 22:10:23.113402 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:23.113374 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npxnl\" (UniqueName: \"kubernetes.io/projected/7c0902f1-2ad0-4787-986f-3519210cb168-kube-api-access-npxnl\") pod \"maas-api-revoke-sub-5702beb6-gcxfv\" (UID: \"7c0902f1-2ad0-4787-986f-3519210cb168\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv" Oct 30 22:10:23.115214 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:23.115196 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv" Oct 30 22:10:23.484193 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:23.484128 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh" Oct 30 22:10:23.484322 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:23.484128 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh" event={"ID":"dcc083f7-11a5-476a-8124-5d38fc2b0468","Type":"ContainerDied","Data":"dbfc674298ee423dbea89cf3d71e6cda7089060cef2118b20f570b594bbbe5db"} Oct 30 22:10:23.484322 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:23.484246 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbfc674298ee423dbea89cf3d71e6cda7089060cef2118b20f570b594bbbe5db" Oct 30 22:10:23.502087 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:10:23.502060 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0902f1_2ad0_4787_986f_3519210cb168.slice/crio-3c51680d6526bb97f685e47bf8bf77bcc1e19bab3ca7023c6462ab734c5de752 WatchSource:0}: Error finding container 3c51680d6526bb97f685e47bf8bf77bcc1e19bab3ca7023c6462ab734c5de752: Status 404 returned error can't find the container with id 3c51680d6526bb97f685e47bf8bf77bcc1e19bab3ca7023c6462ab734c5de752 Oct 30 22:10:23.504254 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:23.504229 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv"] Oct 30 22:10:24.496630 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:24.496586 4335 generic.go:356] "Generic (PLEG): container finished" podID="7c0902f1-2ad0-4787-986f-3519210cb168" containerID="ff1dfbab03d60cd1377d7b9b7dca4c29fdd74c24eeefc614259ddeb6af3f3375" exitCode=0 Oct 30 22:10:24.497072 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:24.496678 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv" event={"ID":"7c0902f1-2ad0-4787-986f-3519210cb168","Type":"ContainerDied","Data":"ff1dfbab03d60cd1377d7b9b7dca4c29fdd74c24eeefc614259ddeb6af3f3375"} Oct 30 22:10:24.497072 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:24.496745 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv" event={"ID":"7c0902f1-2ad0-4787-986f-3519210cb168","Type":"ContainerStarted","Data":"3c51680d6526bb97f685e47bf8bf77bcc1e19bab3ca7023c6462ab734c5de752"} Oct 30 22:10:25.322176 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:10:25.322138 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:10:25.664529 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:25.664505 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv" Oct 30 22:10:25.787060 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:25.787026 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c0902f1-2ad0-4787-986f-3519210cb168-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c0902f1-2ad0-4787-986f-3519210cb168-maas-api-xks-ca\") pod \"7c0902f1-2ad0-4787-986f-3519210cb168\" (UID: \"7c0902f1-2ad0-4787-986f-3519210cb168\") " Oct 30 22:10:25.787236 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:25.787160 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c0902f1-2ad0-4787-986f-3519210cb168-kube-api-access-npxnl\" (UniqueName: \"kubernetes.io/projected/7c0902f1-2ad0-4787-986f-3519210cb168-kube-api-access-npxnl\") pod \"7c0902f1-2ad0-4787-986f-3519210cb168\" (UID: \"7c0902f1-2ad0-4787-986f-3519210cb168\") " Oct 30 22:10:25.787318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:25.787302 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c0902f1-2ad0-4787-986f-3519210cb168-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c0902f1-2ad0-4787-986f-3519210cb168-maas-api-openshift-ca\") pod \"7c0902f1-2ad0-4787-986f-3519210cb168\" (UID: \"7c0902f1-2ad0-4787-986f-3519210cb168\") " Oct 30 22:10:25.787797 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:25.787702 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c0902f1-2ad0-4787-986f-3519210cb168-maas-api-openshift-ca" pod "7c0902f1-2ad0-4787-986f-3519210cb168" (UID: "7c0902f1-2ad0-4787-986f-3519210cb168"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:25.788631 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:25.788586 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c0902f1-2ad0-4787-986f-3519210cb168-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:25.800669 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:25.790137 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c0902f1-2ad0-4787-986f-3519210cb168-kube-api-access-npxnl" pod "7c0902f1-2ad0-4787-986f-3519210cb168" (UID: "7c0902f1-2ad0-4787-986f-3519210cb168"). InnerVolumeSpecName "kube-api-access-npxnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:25.800669 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:25.790183 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c0902f1-2ad0-4787-986f-3519210cb168-maas-api-xks-ca" pod "7c0902f1-2ad0-4787-986f-3519210cb168" (UID: "7c0902f1-2ad0-4787-986f-3519210cb168"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:25.890211 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:25.890154 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c0902f1-2ad0-4787-986f-3519210cb168-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:25.890211 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:25.890178 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npxnl\" (UniqueName: \"kubernetes.io/projected/7c0902f1-2ad0-4787-986f-3519210cb168-kube-api-access-npxnl\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:26.516185 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:26.516160 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv" Oct 30 22:10:26.516343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:26.516219 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv" event={"ID":"7c0902f1-2ad0-4787-986f-3519210cb168","Type":"ContainerDied","Data":"3c51680d6526bb97f685e47bf8bf77bcc1e19bab3ca7023c6462ab734c5de752"} Oct 30 22:10:26.516343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:26.516268 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c51680d6526bb97f685e47bf8bf77bcc1e19bab3ca7023c6462ab734c5de752" Oct 30 22:10:27.617040 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:27.617007 4335 scope.go:122] "RemoveContainer" containerID="db1ed4f9fa1d644cb2bb9d35c0ab821bf7444f6d9f8b6edc77b31d57d9347a8f" Oct 30 22:10:27.629046 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:27.629022 4335 scope.go:122] "RemoveContainer" containerID="fc8f57d2908ec6717dec492a241bc5114877830363686518511698f9dc8ef6f4" Oct 30 22:10:27.639553 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:27.639535 4335 scope.go:122] "RemoveContainer" containerID="bf31fae34bf69693d1b88b4875a039475d7a8decf00363d688e6e4c37cdf85ea" Oct 30 22:10:27.649677 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:27.649657 4335 scope.go:122] "RemoveContainer" containerID="538bfd9ccef295a1709afa55728622bf11c257d83649e6639c7879139fd39938" Oct 30 22:10:27.661979 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:27.661957 4335 scope.go:122] "RemoveContainer" containerID="0a221fa58522dd429a6c5072e1e77343374c2ad3a3d9daf3353c85da914b57a7" Oct 30 22:10:27.672082 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:27.672062 4335 scope.go:122] "RemoveContainer" containerID="a511ad002a3edf5f56c68c4a1054db33f7da6e215759026dd947a2562cce5f79" Oct 30 22:10:27.682542 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:27.681861 4335 scope.go:122] "RemoveContainer" containerID="9741a25ef70beb6bf8f5cd1b2067b63e4a7040478604963e01b3dfdd4bdfd08d" Oct 30 22:10:27.695339 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:27.695316 4335 scope.go:122] "RemoveContainer" containerID="eee024623e7de2c3e50770a268a31bcfa3daced2b57df4dec0d5d92a5715bc5a" Oct 30 22:10:27.705605 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:27.705589 4335 scope.go:122] "RemoveContainer" containerID="070039490cf57217a10b494340f0b2155542e106ebab3bfd0579a0c9546b6c4c" Oct 30 22:10:27.719694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:27.719675 4335 scope.go:122] "RemoveContainer" containerID="f9b191e4bc29ffa127ea6a8ece309f7f013efedf468bfca4e2ba9ee6384b2615" Oct 30 22:10:27.729805 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:27.729786 4335 scope.go:122] "RemoveContainer" containerID="60641eadffad410c29faf7694b68be1d7b50a193507c0e3cff103bfc92def616" Oct 30 22:10:27.742644 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:27.742590 4335 scope.go:122] "RemoveContainer" containerID="e459b3223b4cb40b98a1df7fc07df0042f32e56c99633f37095024a269c0281b" Oct 30 22:10:29.630865 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.630825 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf"] Oct 30 22:10:29.641785 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.639331 4335 cpu_manager.go:414] "containerMap: removing container" podUID="7c0902f1-2ad0-4787-986f-3519210cb168" containerName="revoke-keys" Oct 30 22:10:29.641785 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.639362 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c0902f1-2ad0-4787-986f-3519210cb168" containerName="revoke-keys" Oct 30 22:10:29.641785 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.639597 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c0902f1-2ad0-4787-986f-3519210cb168" containerName="revoke-keys" Oct 30 22:10:29.648868 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.648840 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.666522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.666494 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf"] Oct 30 22:10:29.701356 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.701311 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-92ce375b-gw-openshift-default-dockercfg-qmjtr\"" Oct 30 22:10:29.816536 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.816498 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-credential-socket\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.816758 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.816560 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-token\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.816758 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.816666 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istiod-ca-cert\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.816758 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.816722 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-workload-certs\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.816936 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.816804 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9zlf\" (UniqueName: \"kubernetes.io/projected/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-kube-api-access-g9zlf\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.816936 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.816884 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-podinfo\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.816936 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.816917 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-workload-socket\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.817088 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.817009 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-envoy\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.817088 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.817061 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-data\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.918522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.918433 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9zlf\" (UniqueName: \"kubernetes.io/projected/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-kube-api-access-g9zlf\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.918522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.918504 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-podinfo\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.918829 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.918528 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-workload-socket\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.918829 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.918601 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-envoy\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.918829 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.918672 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-data\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.919021 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.918855 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-credential-socket\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.919021 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.918881 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-token\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.919021 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.918935 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istiod-ca-cert\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.919021 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.919000 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-workload-certs\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.919274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.919246 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-workload-socket\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.919424 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.919396 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-credential-socket\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.919764 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.919741 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-workload-certs\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.919764 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.919750 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-data\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.920284 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.920263 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istiod-ca-cert\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.921349 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.921324 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-envoy\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.921728 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.921708 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-podinfo\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.979026 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.978996 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-token\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:29.979162 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:29.979031 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9zlf\" (UniqueName: \"kubernetes.io/projected/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-kube-api-access-g9zlf\") pod \"e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:30.271748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.271680 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:30.330273 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.330222 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm"] Oct 30 22:10:30.336986 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.336956 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm" Oct 30 22:10:30.391074 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.391041 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:10:30.391269 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.391198 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:10:30.391364 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.391279 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:30.392090 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.392066 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:10:30.403265 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.403216 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm"] Oct 30 22:10:30.428367 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.428293 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-475lc\" (UniqueName: \"kubernetes.io/projected/900d2447-11f1-4eab-95a8-ffca6af2010c-kube-api-access-475lc\") pod \"maas-api-revoke-sub-fa7f9b1e-dkplm\" (UID: \"900d2447-11f1-4eab-95a8-ffca6af2010c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm" Oct 30 22:10:30.428514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.428494 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900d2447-11f1-4eab-95a8-ffca6af2010c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa7f9b1e-dkplm\" (UID: \"900d2447-11f1-4eab-95a8-ffca6af2010c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm" Oct 30 22:10:30.428889 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.428854 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900d2447-11f1-4eab-95a8-ffca6af2010c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa7f9b1e-dkplm\" (UID: \"900d2447-11f1-4eab-95a8-ffca6af2010c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm" Oct 30 22:10:30.531155 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.531079 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-475lc\" (UniqueName: \"kubernetes.io/projected/900d2447-11f1-4eab-95a8-ffca6af2010c-kube-api-access-475lc\") pod \"maas-api-revoke-sub-fa7f9b1e-dkplm\" (UID: \"900d2447-11f1-4eab-95a8-ffca6af2010c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm" Oct 30 22:10:30.531265 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.531215 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900d2447-11f1-4eab-95a8-ffca6af2010c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa7f9b1e-dkplm\" (UID: \"900d2447-11f1-4eab-95a8-ffca6af2010c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm" Oct 30 22:10:30.531318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.531279 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900d2447-11f1-4eab-95a8-ffca6af2010c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa7f9b1e-dkplm\" (UID: \"900d2447-11f1-4eab-95a8-ffca6af2010c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm" Oct 30 22:10:30.531834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.531817 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900d2447-11f1-4eab-95a8-ffca6af2010c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa7f9b1e-dkplm\" (UID: \"900d2447-11f1-4eab-95a8-ffca6af2010c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm" Oct 30 22:10:30.534008 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.533984 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900d2447-11f1-4eab-95a8-ffca6af2010c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa7f9b1e-dkplm\" (UID: \"900d2447-11f1-4eab-95a8-ffca6af2010c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm" Oct 30 22:10:30.592199 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.592167 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-475lc\" (UniqueName: \"kubernetes.io/projected/900d2447-11f1-4eab-95a8-ffca6af2010c-kube-api-access-475lc\") pod \"maas-api-revoke-sub-fa7f9b1e-dkplm\" (UID: \"900d2447-11f1-4eab-95a8-ffca6af2010c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm" Oct 30 22:10:30.672565 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.672436 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm" Oct 30 22:10:30.717449 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.717424 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf"] Oct 30 22:10:30.719044 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:10:30.719020 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27098ffe_5ffe_4d80_bbe3_d4cd7bda73c0.slice/crio-961814a7620a326b7f0ecaaf48b3d8524247128c1968503d209061263e504a2c WatchSource:0}: Error finding container 961814a7620a326b7f0ecaaf48b3d8524247128c1968503d209061263e504a2c: Status 404 returned error can't find the container with id 961814a7620a326b7f0ecaaf48b3d8524247128c1968503d209061263e504a2c Oct 30 22:10:30.721922 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.721885 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:10:30.722045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.721970 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:10:30.722045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:30.722017 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:10:31.067305 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:10:31.067263 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900d2447_11f1_4eab_95a8_ffca6af2010c.slice/crio-c5abf422bb0c7522481fa5e402e7eb780be1ebceb0909e5c086749d6f2798867 WatchSource:0}: Error finding container c5abf422bb0c7522481fa5e402e7eb780be1ebceb0909e5c086749d6f2798867: Status 404 returned error can't find the container with id c5abf422bb0c7522481fa5e402e7eb780be1ebceb0909e5c086749d6f2798867 Oct 30 22:10:31.068182 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:31.068158 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm"] Oct 30 22:10:31.232638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:31.232128 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6954-bc401f"] Oct 30 22:10:31.238636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:31.232948 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6954-bc401f"] Oct 30 22:10:31.238636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:31.233169 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6954-bc401f" podUID="7fd7f856-45c5-4a3c-8622-c15c66fb7f90" containerName="test-curl-6954-bc401f" containerID="cri-o://df85b16372adf38100e3cb851afd50866637c8db3699225fd164d67263119659" gracePeriod=2 Oct 30 22:10:31.284110 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:10:31.284069 4335 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6954-bc401f\" is forbidden: User \"system:node:ip-10-0-131-193.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-193.us-west-1.compute.internal' and this object" podUID="7fd7f856-45c5-4a3c-8622-c15c66fb7f90" pod="opendatahub/test-curl-6954-bc401f" Oct 30 22:10:31.679644 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:31.679572 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" event={"ID":"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0","Type":"ContainerStarted","Data":"c379493889432c301d6e169ec821fe5a8d11569eff5d73932e5176d44ad50c51"} Oct 30 22:10:31.680121 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:31.679651 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" event={"ID":"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0","Type":"ContainerStarted","Data":"961814a7620a326b7f0ecaaf48b3d8524247128c1968503d209061263e504a2c"} Oct 30 22:10:31.681762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:31.681737 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm" event={"ID":"900d2447-11f1-4eab-95a8-ffca6af2010c","Type":"ContainerStarted","Data":"8b18918dadca3c336c53ffe48251244f7169eda446db1a666c2a20cb0e551b07"} Oct 30 22:10:31.681879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:31.681781 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm" event={"ID":"900d2447-11f1-4eab-95a8-ffca6af2010c","Type":"ContainerStarted","Data":"c5abf422bb0c7522481fa5e402e7eb780be1ebceb0909e5c086749d6f2798867"} Oct 30 22:10:31.740438 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:10:31.740381 4335 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6954-bc401f\" is forbidden: User \"system:node:ip-10-0-131-193.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-193.us-west-1.compute.internal' and this object" podUID="7fd7f856-45c5-4a3c-8622-c15c66fb7f90" pod="opendatahub/test-curl-6954-bc401f" Oct 30 22:10:31.865014 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:31.864920 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" podStartSLOduration=2.864905647 podStartE2EDuration="2.864905647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:31.859412558 +0000 UTC m=+1564.959844300" watchObservedRunningTime="2026-10-30 22:10:31.864905647 +0000 UTC m=+1564.965337378" Oct 30 22:10:31.975430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:31.975375 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm" podStartSLOduration=1.9753611599999998 podStartE2EDuration="1.97536116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:31.973419592 +0000 UTC m=+1565.073851327" watchObservedRunningTime="2026-10-30 22:10:31.97536116 +0000 UTC m=+1565.075792892" Oct 30 22:10:32.024147 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:10:32.024100 4335 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6954-bc401f\" is forbidden: User \"system:node:ip-10-0-131-193.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-193.us-west-1.compute.internal' and this object" podUID="7fd7f856-45c5-4a3c-8622-c15c66fb7f90" pod="opendatahub/test-curl-6954-bc401f" Oct 30 22:10:32.272296 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:32.272267 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:32.278131 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:32.278107 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:32.447608 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:10:32.447544 4335 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6954-bc401f\" is forbidden: User \"system:node:ip-10-0-131-193.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-193.us-west-1.compute.internal' and this object" podUID="7fd7f856-45c5-4a3c-8622-c15c66fb7f90" pod="opendatahub/test-curl-6954-bc401f" Oct 30 22:10:32.693605 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:32.693566 4335 generic.go:356] "Generic (PLEG): container finished" podID="900d2447-11f1-4eab-95a8-ffca6af2010c" containerID="8b18918dadca3c336c53ffe48251244f7169eda446db1a666c2a20cb0e551b07" exitCode=0 Oct 30 22:10:32.694054 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:32.693872 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm" event={"ID":"900d2447-11f1-4eab-95a8-ffca6af2010c","Type":"ContainerDied","Data":"8b18918dadca3c336c53ffe48251244f7169eda446db1a666c2a20cb0e551b07"} Oct 30 22:10:32.694167 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:32.694144 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:32.695050 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:32.695031 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:10:32.859126 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:10:32.859091 4335 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6954-bc401f\" is forbidden: User \"system:node:ip-10-0-131-193.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-193.us-west-1.compute.internal' and this object" podUID="7fd7f856-45c5-4a3c-8622-c15c66fb7f90" pod="opendatahub/test-curl-6954-bc401f" Oct 30 22:10:33.038181 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:10:33.038137 4335 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6954-bc401f\" is forbidden: User \"system:node:ip-10-0-131-193.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-193.us-west-1.compute.internal' and this object" podUID="7fd7f856-45c5-4a3c-8622-c15c66fb7f90" pod="opendatahub/test-curl-6954-bc401f" Oct 30 22:10:33.495261 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.495202 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6954-bc401f" Oct 30 22:10:33.581320 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.581290 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fd7f856-45c5-4a3c-8622-c15c66fb7f90-kube-api-access-7cfgp\" (UniqueName: \"kubernetes.io/projected/7fd7f856-45c5-4a3c-8622-c15c66fb7f90-kube-api-access-7cfgp\") pod \"7fd7f856-45c5-4a3c-8622-c15c66fb7f90\" (UID: \"7fd7f856-45c5-4a3c-8622-c15c66fb7f90\") " Oct 30 22:10:33.582385 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.582356 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf"] Oct 30 22:10:33.583504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.583477 4335 cpu_manager.go:414] "containerMap: removing container" podUID="7fd7f856-45c5-4a3c-8622-c15c66fb7f90" containerName="test-curl-6954-bc401f" Oct 30 22:10:33.583588 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.583509 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fd7f856-45c5-4a3c-8622-c15c66fb7f90" containerName="test-curl-6954-bc401f" Oct 30 22:10:33.583588 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.583486 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd7f856-45c5-4a3c-8622-c15c66fb7f90-kube-api-access-7cfgp" pod "7fd7f856-45c5-4a3c-8622-c15c66fb7f90" (UID: "7fd7f856-45c5-4a3c-8622-c15c66fb7f90"). InnerVolumeSpecName "kube-api-access-7cfgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:33.583737 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.583725 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fd7f856-45c5-4a3c-8622-c15c66fb7f90" containerName="test-curl-6954-bc401f" Oct 30 22:10:33.588318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.588278 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf" Oct 30 22:10:33.642666 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.642632 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf"] Oct 30 22:10:33.683740 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.683706 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wgb7\" (UniqueName: \"kubernetes.io/projected/80a004d8-e86c-4151-9a8b-eb208f8f0503-kube-api-access-8wgb7\") pod \"maas-api-revoke-sub-94a4871c-lm2nf\" (UID: \"80a004d8-e86c-4151-9a8b-eb208f8f0503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf" Oct 30 22:10:33.683944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.683917 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80a004d8-e86c-4151-9a8b-eb208f8f0503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94a4871c-lm2nf\" (UID: \"80a004d8-e86c-4151-9a8b-eb208f8f0503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf" Oct 30 22:10:33.684028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.684013 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80a004d8-e86c-4151-9a8b-eb208f8f0503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94a4871c-lm2nf\" (UID: \"80a004d8-e86c-4151-9a8b-eb208f8f0503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf" Oct 30 22:10:33.684292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.684271 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cfgp\" (UniqueName: \"kubernetes.io/projected/7fd7f856-45c5-4a3c-8622-c15c66fb7f90-kube-api-access-7cfgp\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.710850 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.710813 4335 generic.go:356] "Generic (PLEG): container finished" podID="7fd7f856-45c5-4a3c-8622-c15c66fb7f90" containerID="df85b16372adf38100e3cb851afd50866637c8db3699225fd164d67263119659" exitCode=137 Oct 30 22:10:33.711266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.710853 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6954-bc401f" Oct 30 22:10:33.711266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.710905 4335 scope.go:122] "RemoveContainer" containerID="df85b16372adf38100e3cb851afd50866637c8db3699225fd164d67263119659" Oct 30 22:10:33.726103 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.726081 4335 scope.go:122] "RemoveContainer" containerID="df85b16372adf38100e3cb851afd50866637c8db3699225fd164d67263119659" Oct 30 22:10:33.726382 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:10:33.726363 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df85b16372adf38100e3cb851afd50866637c8db3699225fd164d67263119659\": container with ID starting with df85b16372adf38100e3cb851afd50866637c8db3699225fd164d67263119659 not found: ID does not exist" containerID="df85b16372adf38100e3cb851afd50866637c8db3699225fd164d67263119659" Oct 30 22:10:33.726467 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.726391 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df85b16372adf38100e3cb851afd50866637c8db3699225fd164d67263119659"} err="failed to get container status \"df85b16372adf38100e3cb851afd50866637c8db3699225fd164d67263119659\": rpc error: code = NotFound desc = could not find container \"df85b16372adf38100e3cb851afd50866637c8db3699225fd164d67263119659\": container with ID starting with df85b16372adf38100e3cb851afd50866637c8db3699225fd164d67263119659 not found: ID does not exist" Oct 30 22:10:33.772385 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:10:33.772310 4335 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6954-bc401f\" is forbidden: User \"system:node:ip-10-0-131-193.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-193.us-west-1.compute.internal' and this object" podUID="7fd7f856-45c5-4a3c-8622-c15c66fb7f90" pod="opendatahub/test-curl-6954-bc401f" Oct 30 22:10:33.787581 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.787545 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wgb7\" (UniqueName: \"kubernetes.io/projected/80a004d8-e86c-4151-9a8b-eb208f8f0503-kube-api-access-8wgb7\") pod \"maas-api-revoke-sub-94a4871c-lm2nf\" (UID: \"80a004d8-e86c-4151-9a8b-eb208f8f0503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf" Oct 30 22:10:33.788668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.787767 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80a004d8-e86c-4151-9a8b-eb208f8f0503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94a4871c-lm2nf\" (UID: \"80a004d8-e86c-4151-9a8b-eb208f8f0503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf" Oct 30 22:10:33.788668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.787828 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80a004d8-e86c-4151-9a8b-eb208f8f0503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94a4871c-lm2nf\" (UID: \"80a004d8-e86c-4151-9a8b-eb208f8f0503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf" Oct 30 22:10:33.790221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.790199 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80a004d8-e86c-4151-9a8b-eb208f8f0503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94a4871c-lm2nf\" (UID: \"80a004d8-e86c-4151-9a8b-eb208f8f0503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf" Oct 30 22:10:33.801059 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.797072 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80a004d8-e86c-4151-9a8b-eb208f8f0503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94a4871c-lm2nf\" (UID: \"80a004d8-e86c-4151-9a8b-eb208f8f0503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf" Oct 30 22:10:33.849716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.849689 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wgb7\" (UniqueName: \"kubernetes.io/projected/80a004d8-e86c-4151-9a8b-eb208f8f0503-kube-api-access-8wgb7\") pod \"maas-api-revoke-sub-94a4871c-lm2nf\" (UID: \"80a004d8-e86c-4151-9a8b-eb208f8f0503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf" Oct 30 22:10:33.870177 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.870155 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm" Oct 30 22:10:33.889370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.889341 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/900d2447-11f1-4eab-95a8-ffca6af2010c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900d2447-11f1-4eab-95a8-ffca6af2010c-maas-api-xks-ca\") pod \"900d2447-11f1-4eab-95a8-ffca6af2010c\" (UID: \"900d2447-11f1-4eab-95a8-ffca6af2010c\") " Oct 30 22:10:33.889547 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.889529 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900d2447-11f1-4eab-95a8-ffca6af2010c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900d2447-11f1-4eab-95a8-ffca6af2010c-maas-api-openshift-ca\") pod \"900d2447-11f1-4eab-95a8-ffca6af2010c\" (UID: \"900d2447-11f1-4eab-95a8-ffca6af2010c\") " Oct 30 22:10:33.889675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.889576 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/900d2447-11f1-4eab-95a8-ffca6af2010c-kube-api-access-475lc\" (UniqueName: \"kubernetes.io/projected/900d2447-11f1-4eab-95a8-ffca6af2010c-kube-api-access-475lc\") pod \"900d2447-11f1-4eab-95a8-ffca6af2010c\" (UID: \"900d2447-11f1-4eab-95a8-ffca6af2010c\") " Oct 30 22:10:33.889969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.889946 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900d2447-11f1-4eab-95a8-ffca6af2010c-maas-api-openshift-ca" pod "900d2447-11f1-4eab-95a8-ffca6af2010c" (UID: "900d2447-11f1-4eab-95a8-ffca6af2010c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:33.890208 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.890187 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900d2447-11f1-4eab-95a8-ffca6af2010c-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.891779 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.891749 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/900d2447-11f1-4eab-95a8-ffca6af2010c-maas-api-xks-ca" pod "900d2447-11f1-4eab-95a8-ffca6af2010c" (UID: "900d2447-11f1-4eab-95a8-ffca6af2010c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:33.892212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.892193 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900d2447-11f1-4eab-95a8-ffca6af2010c-kube-api-access-475lc" pod "900d2447-11f1-4eab-95a8-ffca6af2010c" (UID: "900d2447-11f1-4eab-95a8-ffca6af2010c"). InnerVolumeSpecName "kube-api-access-475lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:33.915847 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.915823 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf" Oct 30 22:10:33.920816 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:10:33.920788 4335 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6954-bc401f\" is forbidden: User \"system:node:ip-10-0-131-193.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-193.us-west-1.compute.internal' and this object" podUID="7fd7f856-45c5-4a3c-8622-c15c66fb7f90" pod="opendatahub/test-curl-6954-bc401f" Oct 30 22:10:33.992455 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.992426 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-475lc\" (UniqueName: \"kubernetes.io/projected/900d2447-11f1-4eab-95a8-ffca6af2010c-kube-api-access-475lc\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:33.992571 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:33.992457 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900d2447-11f1-4eab-95a8-ffca6af2010c-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:34.305501 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:10:34.305466 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a004d8_e86c_4151_9a8b_eb208f8f0503.slice/crio-fee2b780afa14664c8f54a5fc83cd867f27509f2c4f295fd5cee6c578e45777d WatchSource:0}: Error finding container fee2b780afa14664c8f54a5fc83cd867f27509f2c4f295fd5cee6c578e45777d: Status 404 returned error can't find the container with id fee2b780afa14664c8f54a5fc83cd867f27509f2c4f295fd5cee6c578e45777d Oct 30 22:10:34.307972 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:34.307944 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf"] Oct 30 22:10:34.723019 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:34.722993 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm" Oct 30 22:10:34.723392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:34.722994 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm" event={"ID":"900d2447-11f1-4eab-95a8-ffca6af2010c","Type":"ContainerDied","Data":"c5abf422bb0c7522481fa5e402e7eb780be1ebceb0909e5c086749d6f2798867"} Oct 30 22:10:34.723392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:34.723115 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5abf422bb0c7522481fa5e402e7eb780be1ebceb0909e5c086749d6f2798867" Oct 30 22:10:34.724672 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:34.724648 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf" event={"ID":"80a004d8-e86c-4151-9a8b-eb208f8f0503","Type":"ContainerStarted","Data":"fee2b780afa14664c8f54a5fc83cd867f27509f2c4f295fd5cee6c578e45777d"} Oct 30 22:10:34.834827 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:10:34.834784 4335 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6954-bc401f\" is forbidden: User \"system:node:ip-10-0-131-193.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-193.us-west-1.compute.internal' and this object" podUID="7fd7f856-45c5-4a3c-8622-c15c66fb7f90" pod="opendatahub/test-curl-6954-bc401f" Oct 30 22:10:35.060213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:35.060180 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fd7f856-45c5-4a3c-8622-c15c66fb7f90" path="/var/lib/kubelet/pods/7fd7f856-45c5-4a3c-8622-c15c66fb7f90/volumes" Oct 30 22:10:35.738854 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:35.738765 4335 generic.go:356] "Generic (PLEG): container finished" podID="80a004d8-e86c-4151-9a8b-eb208f8f0503" containerID="2115741a634c367e939ba61aaf910c447d27efcd8b91b3ff67611948ee1ba263" exitCode=0 Oct 30 22:10:35.738854 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:35.738835 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf" event={"ID":"80a004d8-e86c-4151-9a8b-eb208f8f0503","Type":"ContainerDied","Data":"2115741a634c367e939ba61aaf910c447d27efcd8b91b3ff67611948ee1ba263"} Oct 30 22:10:36.399294 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.399259 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp"] Oct 30 22:10:36.400090 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.400072 4335 cpu_manager.go:414] "containerMap: removing container" podUID="900d2447-11f1-4eab-95a8-ffca6af2010c" containerName="revoke-keys" Oct 30 22:10:36.400156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.400091 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="900d2447-11f1-4eab-95a8-ffca6af2010c" containerName="revoke-keys" Oct 30 22:10:36.400343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.400326 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="900d2447-11f1-4eab-95a8-ffca6af2010c" containerName="revoke-keys" Oct 30 22:10:36.405498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.405478 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp" Oct 30 22:10:36.460672 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.460215 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2678f4e1\"" Oct 30 22:10:36.472587 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.472169 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp"] Oct 30 22:10:36.501289 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.501262 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2678f4e1-dockercfg-v98p7\"" Oct 30 22:10:36.612683 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.605084 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67660558-c6ab-4de8-8f30-80b80c466214-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp\" (UID: \"67660558-c6ab-4de8-8f30-80b80c466214\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp" Oct 30 22:10:36.612683 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.606083 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwr56\" (UniqueName: \"kubernetes.io/projected/67660558-c6ab-4de8-8f30-80b80c466214-kube-api-access-rwr56\") pod \"payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp\" (UID: \"67660558-c6ab-4de8-8f30-80b80c466214\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp" Oct 30 22:10:36.652785 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.652690 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2"] Oct 30 22:10:36.665924 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.665896 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2" Oct 30 22:10:36.668447 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.668410 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2"] Oct 30 22:10:36.717656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.710364 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67660558-c6ab-4de8-8f30-80b80c466214-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp\" (UID: \"67660558-c6ab-4de8-8f30-80b80c466214\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp" Oct 30 22:10:36.717656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.710423 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwr56\" (UniqueName: \"kubernetes.io/projected/67660558-c6ab-4de8-8f30-80b80c466214-kube-api-access-rwr56\") pod \"payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp\" (UID: \"67660558-c6ab-4de8-8f30-80b80c466214\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp" Oct 30 22:10:36.717656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.711700 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67660558-c6ab-4de8-8f30-80b80c466214-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp\" (UID: \"67660558-c6ab-4de8-8f30-80b80c466214\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp" Oct 30 22:10:36.774441 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.773693 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwr56\" (UniqueName: \"kubernetes.io/projected/67660558-c6ab-4de8-8f30-80b80c466214-kube-api-access-rwr56\") pod \"payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp\" (UID: \"67660558-c6ab-4de8-8f30-80b80c466214\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp" Oct 30 22:10:36.812006 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.811970 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w27tk\" (UniqueName: \"kubernetes.io/projected/5b3987ee-87d7-445f-a49a-9d62ea088f60-kube-api-access-w27tk\") pod \"payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2\" (UID: \"5b3987ee-87d7-445f-a49a-9d62ea088f60\") " pod="openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2" Oct 30 22:10:36.812193 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.812120 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3987ee-87d7-445f-a49a-9d62ea088f60-plugins-config-volume\") pod \"payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2\" (UID: \"5b3987ee-87d7-445f-a49a-9d62ea088f60\") " pod="openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2" Oct 30 22:10:36.913768 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.913680 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3987ee-87d7-445f-a49a-9d62ea088f60-plugins-config-volume\") pod \"payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2\" (UID: \"5b3987ee-87d7-445f-a49a-9d62ea088f60\") " pod="openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2" Oct 30 22:10:36.913944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.913909 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w27tk\" (UniqueName: \"kubernetes.io/projected/5b3987ee-87d7-445f-a49a-9d62ea088f60-kube-api-access-w27tk\") pod \"payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2\" (UID: \"5b3987ee-87d7-445f-a49a-9d62ea088f60\") " pod="openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2" Oct 30 22:10:36.914944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.914913 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3987ee-87d7-445f-a49a-9d62ea088f60-plugins-config-volume\") pod \"payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2\" (UID: \"5b3987ee-87d7-445f-a49a-9d62ea088f60\") " pod="openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2" Oct 30 22:10:36.952659 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.952609 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf" Oct 30 22:10:36.974792 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:36.974760 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w27tk\" (UniqueName: \"kubernetes.io/projected/5b3987ee-87d7-445f-a49a-9d62ea088f60-kube-api-access-w27tk\") pod \"payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2\" (UID: \"5b3987ee-87d7-445f-a49a-9d62ea088f60\") " pod="openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2" Oct 30 22:10:37.015125 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.015099 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80a004d8-e86c-4151-9a8b-eb208f8f0503-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80a004d8-e86c-4151-9a8b-eb208f8f0503-maas-api-openshift-ca\") pod \"80a004d8-e86c-4151-9a8b-eb208f8f0503\" (UID: \"80a004d8-e86c-4151-9a8b-eb208f8f0503\") " Oct 30 22:10:37.015234 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.015157 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a004d8-e86c-4151-9a8b-eb208f8f0503-kube-api-access-8wgb7\" (UniqueName: \"kubernetes.io/projected/80a004d8-e86c-4151-9a8b-eb208f8f0503-kube-api-access-8wgb7\") pod \"80a004d8-e86c-4151-9a8b-eb208f8f0503\" (UID: \"80a004d8-e86c-4151-9a8b-eb208f8f0503\") " Oct 30 22:10:37.015306 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.015235 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80a004d8-e86c-4151-9a8b-eb208f8f0503-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80a004d8-e86c-4151-9a8b-eb208f8f0503-maas-api-xks-ca\") pod \"80a004d8-e86c-4151-9a8b-eb208f8f0503\" (UID: \"80a004d8-e86c-4151-9a8b-eb208f8f0503\") " Oct 30 22:10:37.015505 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.015477 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a004d8-e86c-4151-9a8b-eb208f8f0503-maas-api-openshift-ca" pod "80a004d8-e86c-4151-9a8b-eb208f8f0503" (UID: "80a004d8-e86c-4151-9a8b-eb208f8f0503"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:37.016468 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.016003 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80a004d8-e86c-4151-9a8b-eb208f8f0503-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:37.023188 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.023160 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a004d8-e86c-4151-9a8b-eb208f8f0503-kube-api-access-8wgb7" pod "80a004d8-e86c-4151-9a8b-eb208f8f0503" (UID: "80a004d8-e86c-4151-9a8b-eb208f8f0503"). InnerVolumeSpecName "kube-api-access-8wgb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:37.023912 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.023887 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80a004d8-e86c-4151-9a8b-eb208f8f0503-maas-api-xks-ca" pod "80a004d8-e86c-4151-9a8b-eb208f8f0503" (UID: "80a004d8-e86c-4151-9a8b-eb208f8f0503"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:37.025059 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.025034 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp" Oct 30 22:10:37.052149 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.052127 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2" Oct 30 22:10:37.119187 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.117897 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wgb7\" (UniqueName: \"kubernetes.io/projected/80a004d8-e86c-4151-9a8b-eb208f8f0503-kube-api-access-8wgb7\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:37.119187 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.117929 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80a004d8-e86c-4151-9a8b-eb208f8f0503-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:37.420195 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.420116 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp"] Oct 30 22:10:37.451919 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:10:37.451890 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67660558_c6ab_4de8_8f30_80b80c466214.slice/crio-df20ae556449db4c6fdcbc41e7299effff921b5f3d88833f15c45277309dfea6 WatchSource:0}: Error finding container df20ae556449db4c6fdcbc41e7299effff921b5f3d88833f15c45277309dfea6: Status 404 returned error can't find the container with id df20ae556449db4c6fdcbc41e7299effff921b5f3d88833f15c45277309dfea6 Oct 30 22:10:37.472545 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.472517 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2"] Oct 30 22:10:37.472773 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:10:37.472744 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3987ee_87d7_445f_a49a_9d62ea088f60.slice/crio-44bb4d9526177786c39855d4e3ebdcdef0600279c24c6b6c5c2f705c4760469d WatchSource:0}: Error finding container 44bb4d9526177786c39855d4e3ebdcdef0600279c24c6b6c5c2f705c4760469d: Status 404 returned error can't find the container with id 44bb4d9526177786c39855d4e3ebdcdef0600279c24c6b6c5c2f705c4760469d Oct 30 22:10:37.762602 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.762513 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2" event={"ID":"5b3987ee-87d7-445f-a49a-9d62ea088f60","Type":"ContainerStarted","Data":"ba3499ed4177207ad77a436b603315d8678fe15530d1ec67d7a726f56757d365"} Oct 30 22:10:37.762602 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.762573 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2" event={"ID":"5b3987ee-87d7-445f-a49a-9d62ea088f60","Type":"ContainerStarted","Data":"44bb4d9526177786c39855d4e3ebdcdef0600279c24c6b6c5c2f705c4760469d"} Oct 30 22:10:37.762885 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.762605 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2" Oct 30 22:10:37.764888 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.764857 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf" event={"ID":"80a004d8-e86c-4151-9a8b-eb208f8f0503","Type":"ContainerDied","Data":"fee2b780afa14664c8f54a5fc83cd867f27509f2c4f295fd5cee6c578e45777d"} Oct 30 22:10:37.765010 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.764901 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fee2b780afa14664c8f54a5fc83cd867f27509f2c4f295fd5cee6c578e45777d" Oct 30 22:10:37.765010 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.764868 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf" Oct 30 22:10:37.766927 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.766788 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp" event={"ID":"67660558-c6ab-4de8-8f30-80b80c466214","Type":"ContainerStarted","Data":"6d23f27a4c2ff7911a9bcd1bf15afa9b1a7e3fd78e702570c7cb6c5a4486aa64"} Oct 30 22:10:37.766927 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.766852 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp" event={"ID":"67660558-c6ab-4de8-8f30-80b80c466214","Type":"ContainerStarted","Data":"df20ae556449db4c6fdcbc41e7299effff921b5f3d88833f15c45277309dfea6"} Oct 30 22:10:37.766927 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.766882 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp" Oct 30 22:10:37.883780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:37.883717 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2" podStartSLOduration=1.883698391 podStartE2EDuration="1.883698391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:37.878530181 +0000 UTC m=+1570.978961914" watchObservedRunningTime="2026-10-30 22:10:37.883698391 +0000 UTC m=+1570.984130124" Oct 30 22:10:38.056926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:38.053790 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp" podStartSLOduration=2.053770857 podStartE2EDuration="2.053770857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:38.047389261 +0000 UTC m=+1571.147820994" watchObservedRunningTime="2026-10-30 22:10:38.053770857 +0000 UTC m=+1571.154202590" Oct 30 22:10:39.509050 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:39.508306 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq"] Oct 30 22:10:39.509540 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:39.509500 4335 cpu_manager.go:414] "containerMap: removing container" podUID="80a004d8-e86c-4151-9a8b-eb208f8f0503" containerName="revoke-keys" Oct 30 22:10:39.509540 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:39.509521 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a004d8-e86c-4151-9a8b-eb208f8f0503" containerName="revoke-keys" Oct 30 22:10:39.509770 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:39.509750 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a004d8-e86c-4151-9a8b-eb208f8f0503" containerName="revoke-keys" Oct 30 22:10:39.514767 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:39.514736 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq" Oct 30 22:10:39.611650 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:39.610973 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-92ce375b\"" Oct 30 22:10:39.611650 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:39.611131 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-92ce375b-dockercfg-c7gmd\"" Oct 30 22:10:39.666684 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:39.665036 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq"] Oct 30 22:10:39.678199 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:39.677134 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6b9r\" (UniqueName: \"kubernetes.io/projected/368ca2ba-0376-4928-8257-3f9af2de220e-kube-api-access-z6b9r\") pod \"payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq\" (UID: \"368ca2ba-0376-4928-8257-3f9af2de220e\") " pod="openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq" Oct 30 22:10:39.678199 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:39.677283 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/368ca2ba-0376-4928-8257-3f9af2de220e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq\" (UID: \"368ca2ba-0376-4928-8257-3f9af2de220e\") " pod="openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq" Oct 30 22:10:39.779108 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:39.779007 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6b9r\" (UniqueName: \"kubernetes.io/projected/368ca2ba-0376-4928-8257-3f9af2de220e-kube-api-access-z6b9r\") pod \"payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq\" (UID: \"368ca2ba-0376-4928-8257-3f9af2de220e\") " pod="openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq" Oct 30 22:10:39.779291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:39.779141 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/368ca2ba-0376-4928-8257-3f9af2de220e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq\" (UID: \"368ca2ba-0376-4928-8257-3f9af2de220e\") " pod="openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq" Oct 30 22:10:39.779982 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:39.779955 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/368ca2ba-0376-4928-8257-3f9af2de220e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq\" (UID: \"368ca2ba-0376-4928-8257-3f9af2de220e\") " pod="openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq" Oct 30 22:10:39.853085 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:39.852998 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6b9r\" (UniqueName: \"kubernetes.io/projected/368ca2ba-0376-4928-8257-3f9af2de220e-kube-api-access-z6b9r\") pod \"payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq\" (UID: \"368ca2ba-0376-4928-8257-3f9af2de220e\") " pod="openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq" Oct 30 22:10:40.138667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:40.138610 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq" Oct 30 22:10:40.530329 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:10:40.530297 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368ca2ba_0376_4928_8257_3f9af2de220e.slice/crio-47675aefc07f86c08dd2fb2024657f4b24b81964cba9ec49d71f595d1e3fc67e WatchSource:0}: Error finding container 47675aefc07f86c08dd2fb2024657f4b24b81964cba9ec49d71f595d1e3fc67e: Status 404 returned error can't find the container with id 47675aefc07f86c08dd2fb2024657f4b24b81964cba9ec49d71f595d1e3fc67e Oct 30 22:10:40.531405 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:40.531379 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq"] Oct 30 22:10:40.817192 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:40.817153 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq" event={"ID":"368ca2ba-0376-4928-8257-3f9af2de220e","Type":"ContainerStarted","Data":"0b733190948811de462dfcf177a3e677b78696b87c7e1c9d25ee0fe6bb4e5704"} Oct 30 22:10:40.817378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:40.817222 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq" event={"ID":"368ca2ba-0376-4928-8257-3f9af2de220e","Type":"ContainerStarted","Data":"47675aefc07f86c08dd2fb2024657f4b24b81964cba9ec49d71f595d1e3fc67e"} Oct 30 22:10:40.817378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:40.817251 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq" Oct 30 22:10:40.935892 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:40.935831 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq" podStartSLOduration=1.935812452 podStartE2EDuration="1.935812452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:40.931756914 +0000 UTC m=+1574.032188648" watchObservedRunningTime="2026-10-30 22:10:40.935812452 +0000 UTC m=+1574.036244185" Oct 30 22:10:44.107167 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.107137 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:10:44.486583 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.486507 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7"] Oct 30 22:10:44.500092 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.500037 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.552707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.552151 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b8490f04-openshift-default-dockercfg-nglrz\"" Oct 30 22:10:44.609111 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.609079 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7"] Oct 30 22:10:44.644375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.644342 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-credential-socket\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.644551 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.644525 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r29zg\" (UniqueName: \"kubernetes.io/projected/576a247d-5dc2-43f5-9f67-4b572ea9a072-kube-api-access-r29zg\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.644694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.644674 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576a247d-5dc2-43f5-9f67-4b572ea9a072-istiod-ca-cert\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.644768 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.644736 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-podinfo\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.644865 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.644845 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-workload-socket\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.644921 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.644906 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-data\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.644962 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.644942 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-token\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.645093 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.645056 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-envoy\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.645143 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.645128 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-workload-certs\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.770109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.770012 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576a247d-5dc2-43f5-9f67-4b572ea9a072-istiod-ca-cert\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.773107 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.773079 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-podinfo\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.773302 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.773286 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-workload-socket\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.773452 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.773437 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-data\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.773566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.773551 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-token\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.797719 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.797692 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-envoy\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.798319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.798293 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-workload-certs\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.798555 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.798532 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-credential-socket\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.798881 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.798861 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r29zg\" (UniqueName: \"kubernetes.io/projected/576a247d-5dc2-43f5-9f67-4b572ea9a072-kube-api-access-r29zg\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.799196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.774340 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-data\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.799305 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.772356 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576a247d-5dc2-43f5-9f67-4b572ea9a072-istiod-ca-cert\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.799702 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.799682 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-workload-certs\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.799822 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.797228 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-podinfo\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.800160 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.800142 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-credential-socket\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.800268 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.797549 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-workload-socket\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.827636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.815945 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-envoy\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.911638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.911473 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r29zg\" (UniqueName: \"kubernetes.io/projected/576a247d-5dc2-43f5-9f67-4b572ea9a072-kube-api-access-r29zg\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:44.952636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:44.944015 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-token\") pod \"e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:45.121983 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:45.121945 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:45.820182 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:45.819145 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 30 22:10:46.143698 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:46.135960 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Oct 30 22:10:46.224657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:46.217020 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:46.459709 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:46.439475 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:46.459709 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:46.457949 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Oct 30 22:10:46.583641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:46.481073 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:46.632641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:46.525767 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:46.670745 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:46.670700 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:46.692763 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:46.692706 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:46.694336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:46.694302 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 22:10:46.702642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:46.696270 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:46.793439 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:46.793375 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:46.810558 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:46.810522 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7"] Oct 30 22:10:46.822741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:46.822698 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:46.824938 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:46.824914 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:10:46.834263 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:10:46.834235 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576a247d_5dc2_43f5_9f67_4b572ea9a072.slice/crio-5af978520bb017c127bc4e8a8f299bd2acea3c91c047eaf381603c1898b2bc02 WatchSource:0}: Error finding container 5af978520bb017c127bc4e8a8f299bd2acea3c91c047eaf381603c1898b2bc02: Status 404 returned error can't find the container with id 5af978520bb017c127bc4e8a8f299bd2acea3c91c047eaf381603c1898b2bc02 Oct 30 22:10:46.862786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:46.862740 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:10:46.863104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:46.863076 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:10:46.863279 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:46.863258 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:10:46.881042 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:46.881003 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:47.160650 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.160283 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:47.218642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.217666 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 22:10:47.370904 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.370854 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:47.485695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.457937 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:47.485969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.462701 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:47.604791 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.604761 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:10:47.605118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.605100 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:10:47.605355 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.605340 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:10:47.612687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.612664 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:10:47.626538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.626498 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:47.627354 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.627331 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 22:10:47.688835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.652783 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:47.689918 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.671420 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:47.717207 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.691753 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:10:47.747159 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.747078 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:10:47.747943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.697974 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": context deadline exceeded" Oct 30 22:10:47.748388 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.715516 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:47.749565 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.749543 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:10:47.787752 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.749745 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:10:47.788952 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.697942 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:47.789838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.789816 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:10:47.878095 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.874798 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 30 22:10:47.880522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:47.876606 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded" Oct 30 22:10:48.044693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:48.032940 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:48.054505 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:48.054476 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:10:48.688208 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:48.534037 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Oct 30 22:10:48.688208 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:48.573148 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 22:10:48.688208 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:48.627325 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 22:10:48.722932 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:48.722870 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:48.813016 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:48.808930 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:48.818155 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:48.808988 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:48.973576 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:48.973474 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 22:10:48.998888 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:48.998847 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:49.027123 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:49.027078 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 30 22:10:49.075573 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:49.075510 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 30 22:10:49.104921 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:49.104872 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:49.315895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:49.315846 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:49.416677 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:49.353098 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 22:10:49.545677 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:49.539037 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": context deadline exceeded" Oct 30 22:10:49.558753 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:49.558709 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:49.578691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:49.574713 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:49.672644 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:49.640726 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:49.906833 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:49.906732 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": context deadline exceeded" Oct 30 22:10:49.938782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:49.938739 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Oct 30 22:10:49.989767 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:49.989724 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:50.107751 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:50.107696 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:50.137733 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:50.136730 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:50.288696 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:10:50.242172 4335 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.136s" Oct 30 22:10:50.533641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:50.481073 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:10:50.582696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:50.548026 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" event={"ID":"576a247d-5dc2-43f5-9f67-4b572ea9a072","Type":"ContainerStarted","Data":"5af978520bb017c127bc4e8a8f299bd2acea3c91c047eaf381603c1898b2bc02"} Oct 30 22:10:50.582696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:50.548135 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2" Oct 30 22:10:50.582696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:50.548239 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp" Oct 30 22:10:50.671641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:50.421168 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:50.797743 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:50.757973 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:50.797743 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:50.759743 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 22:10:50.797743 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:50.760467 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:50.797743 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:50.786925 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:50.827807 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:50.827769 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:10:50.835813 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:50.835731 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" podUID="6c79ac4e-ddf1-41d0-a8ac-326974828e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:50.859168 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:50.859091 4335 patch_prober.go:29] interesting pod/e2e-worker-w3-cef074-openshift-default-569676869-c9x78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:10:50.865687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:50.859425 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" podUID="d6978eea-f24f-4d30-83b0-85b0015baf53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:50.921667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:50.919297 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:50.950386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:50.950246 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:51.187745 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:51.023922 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Oct 30 22:10:51.194729 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:51.144762 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:51.196534 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:51.169007 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Oct 30 22:10:51.197742 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:51.182663 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:51.481839 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:51.481739 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:51.499628 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:51.499563 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:51.713746 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:51.713698 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 22:10:51.739821 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:51.739730 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:51.741120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:51.741088 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:10:51.741313 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:51.741288 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" podUID="88f88e61-2539-422d-b204-cf54cec014f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:51.776458 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:51.776412 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Oct 30 22:10:51.778026 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:51.777993 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 22:10:51.786645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:51.780441 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:51.787517 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:51.787489 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 30 22:10:51.892642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:51.890761 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:10:51.921643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:51.894674 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" podUID="769e08d0-bf5e-404e-8b7a-adf43dcd0905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:51.922883 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:51.922847 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:52.001856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.001757 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Oct 30 22:10:52.186292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.186234 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" event={"ID":"576a247d-5dc2-43f5-9f67-4b572ea9a072","Type":"ContainerStarted","Data":"ca898a0d633cfa4c00ee3552245837019b7e807ce4742781dbc44c08d7df89a6"} Oct 30 22:10:52.207238 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.207199 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Oct 30 22:10:52.207752 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.207705 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:52.214201 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.214171 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": context deadline exceeded" Oct 30 22:10:52.282918 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.275677 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:10:52.285868 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.285840 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq" Oct 30 22:10:52.528842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.528790 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 22:10:52.657378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.657349 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:10:52.684796 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.684764 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:10:52.715136 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.714421 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:52.756661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.743718 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:52.789540 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.784100 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 22:10:52.792880 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.791326 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:10:52.865369 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.865329 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:52.866045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.787216 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:52.940558 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.940478 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:10:52.974738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.972838 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:10:52.977707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:52.977676 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:53.003519 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:53.003475 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:53.016299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:53.016273 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:10:53.093309 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:53.093277 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:10:53.150640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:53.096593 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:10:53.150640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:53.139878 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:10:53.150640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:53.139923 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:10:53.153297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:53.153192 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" Oct 30 22:10:53.156494 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:53.156474 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:53.157608 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:53.157591 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:10:53.161657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:53.160930 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:53.175403 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:53.175350 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" podStartSLOduration=9.175335262 podStartE2EDuration="9.175335262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:52.700764804 +0000 UTC m=+1585.801196537" watchObservedRunningTime="2026-10-30 22:10:53.175335262 +0000 UTC m=+1586.275766995" Oct 30 22:10:53.195296 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:53.195237 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:53.196192 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:53.196172 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:10:55.326356 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:10:55.326320 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:10:57.339806 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:57.339775 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x"] Oct 30 22:10:57.347424 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:57.347396 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x" Oct 30 22:10:57.403322 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:57.403295 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:57.452925 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:57.452891 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:10:57.453081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:57.452933 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:10:57.453081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:57.452932 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:10:57.459680 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:57.459658 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x"] Oct 30 22:10:57.507694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:57.507667 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9cc2135-q9l2x\" (UID: \"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x" Oct 30 22:10:57.507821 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:57.507787 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9cc2135-q9l2x\" (UID: \"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x" Oct 30 22:10:57.507993 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:57.507973 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9px42\" (UniqueName: \"kubernetes.io/projected/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-kube-api-access-9px42\") pod \"maas-api-revoke-sub-a9cc2135-q9l2x\" (UID: \"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x" Oct 30 22:10:57.609357 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:57.609289 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9cc2135-q9l2x\" (UID: \"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x" Oct 30 22:10:57.609357 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:57.609346 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9cc2135-q9l2x\" (UID: \"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x" Oct 30 22:10:57.609568 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:57.609510 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9px42\" (UniqueName: \"kubernetes.io/projected/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-kube-api-access-9px42\") pod \"maas-api-revoke-sub-a9cc2135-q9l2x\" (UID: \"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x" Oct 30 22:10:57.610017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:57.609995 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9cc2135-q9l2x\" (UID: \"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x" Oct 30 22:10:57.612077 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:57.612056 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9cc2135-q9l2x\" (UID: \"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x" Oct 30 22:10:57.673632 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:57.673593 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9px42\" (UniqueName: \"kubernetes.io/projected/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-kube-api-access-9px42\") pod \"maas-api-revoke-sub-a9cc2135-q9l2x\" (UID: \"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x" Oct 30 22:10:57.963107 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:57.963031 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x" Oct 30 22:10:58.467471 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:10:58.466913 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbf29c0_be44_4c3b_a196_b1af8a4f27b7.slice/crio-0f50b5997ca76373649a6a5dddb67bc6a8fc9f3ee6c5ef929229e5152c463f49 WatchSource:0}: Error finding container 0f50b5997ca76373649a6a5dddb67bc6a8fc9f3ee6c5ef929229e5152c463f49: Status 404 returned error can't find the container with id 0f50b5997ca76373649a6a5dddb67bc6a8fc9f3ee6c5ef929229e5152c463f49 Oct 30 22:10:58.467908 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:58.467774 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x"] Oct 30 22:10:59.433142 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:59.433098 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x" event={"ID":"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7","Type":"ContainerStarted","Data":"7a601301d43106135bb2a635c7596d0b32163ca9777490b9a8f7f992bd79c78a"} Oct 30 22:10:59.433343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:59.433181 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x" event={"ID":"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7","Type":"ContainerStarted","Data":"0f50b5997ca76373649a6a5dddb67bc6a8fc9f3ee6c5ef929229e5152c463f49"} Oct 30 22:10:59.551891 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:10:59.551837 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x" podStartSLOduration=2.5518239339999997 podStartE2EDuration="2.551823934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:59.547671162 +0000 UTC m=+1592.648102894" watchObservedRunningTime="2026-10-30 22:10:59.551823934 +0000 UTC m=+1592.652255668" Oct 30 22:11:00.450224 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:00.450183 4335 generic.go:356] "Generic (PLEG): container finished" podID="4cbf29c0-be44-4c3b-a196-b1af8a4f27b7" containerID="7a601301d43106135bb2a635c7596d0b32163ca9777490b9a8f7f992bd79c78a" exitCode=0 Oct 30 22:11:00.450500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:00.450240 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x" event={"ID":"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7","Type":"ContainerDied","Data":"7a601301d43106135bb2a635c7596d0b32163ca9777490b9a8f7f992bd79c78a"} Oct 30 22:11:01.638033 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:01.638007 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x" Oct 30 22:11:01.773880 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:01.773800 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-maas-api-xks-ca\") pod \"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7\" (UID: \"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7\") " Oct 30 22:11:01.774067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:01.773994 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-maas-api-openshift-ca\") pod \"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7\" (UID: \"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7\") " Oct 30 22:11:01.774157 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:01.774138 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-kube-api-access-9px42\" (UniqueName: \"kubernetes.io/projected/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-kube-api-access-9px42\") pod \"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7\" (UID: \"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7\") " Oct 30 22:11:01.776029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:01.775998 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-maas-api-openshift-ca" pod "4cbf29c0-be44-4c3b-a196-b1af8a4f27b7" (UID: "4cbf29c0-be44-4c3b-a196-b1af8a4f27b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:01.777282 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:01.777253 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-kube-api-access-9px42" pod "4cbf29c0-be44-4c3b-a196-b1af8a4f27b7" (UID: "4cbf29c0-be44-4c3b-a196-b1af8a4f27b7"). InnerVolumeSpecName "kube-api-access-9px42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:01.777716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:01.777693 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-maas-api-xks-ca" pod "4cbf29c0-be44-4c3b-a196-b1af8a4f27b7" (UID: "4cbf29c0-be44-4c3b-a196-b1af8a4f27b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:01.876196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:01.876166 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9px42\" (UniqueName: \"kubernetes.io/projected/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-kube-api-access-9px42\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:01.876196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:01.876196 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:01.876339 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:01.876209 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:02.470904 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:02.470870 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x" Oct 30 22:11:02.471077 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:02.470904 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x" event={"ID":"4cbf29c0-be44-4c3b-a196-b1af8a4f27b7","Type":"ContainerDied","Data":"0f50b5997ca76373649a6a5dddb67bc6a8fc9f3ee6c5ef929229e5152c463f49"} Oct 30 22:11:02.471077 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:02.470952 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f50b5997ca76373649a6a5dddb67bc6a8fc9f3ee6c5ef929229e5152c463f49" Oct 30 22:11:06.394025 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.393992 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d"] Oct 30 22:11:06.394750 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.394732 4335 cpu_manager.go:414] "containerMap: removing container" podUID="4cbf29c0-be44-4c3b-a196-b1af8a4f27b7" containerName="revoke-keys" Oct 30 22:11:06.394750 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.394749 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbf29c0-be44-4c3b-a196-b1af8a4f27b7" containerName="revoke-keys" Oct 30 22:11:06.394890 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.394880 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbf29c0-be44-4c3b-a196-b1af8a4f27b7" containerName="revoke-keys" Oct 30 22:11:06.399064 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.399045 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d" Oct 30 22:11:06.457907 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.457881 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d"] Oct 30 22:11:06.498263 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.498186 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:11:06.498429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.498411 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:11:06.498514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.498461 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:11:06.503173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.503145 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:11:06.535338 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.535310 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56vgz\" (UniqueName: \"kubernetes.io/projected/0955b107-e6d3-495d-9ade-67a6f9ada6bb-kube-api-access-56vgz\") pod \"maas-api-revoke-sub-f22e8244-2qh4d\" (UID: \"0955b107-e6d3-495d-9ade-67a6f9ada6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d" Oct 30 22:11:06.535482 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.535358 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0955b107-e6d3-495d-9ade-67a6f9ada6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22e8244-2qh4d\" (UID: \"0955b107-e6d3-495d-9ade-67a6f9ada6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d" Oct 30 22:11:06.535482 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.535466 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0955b107-e6d3-495d-9ade-67a6f9ada6bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22e8244-2qh4d\" (UID: \"0955b107-e6d3-495d-9ade-67a6f9ada6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d" Oct 30 22:11:06.637303 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.637262 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56vgz\" (UniqueName: \"kubernetes.io/projected/0955b107-e6d3-495d-9ade-67a6f9ada6bb-kube-api-access-56vgz\") pod \"maas-api-revoke-sub-f22e8244-2qh4d\" (UID: \"0955b107-e6d3-495d-9ade-67a6f9ada6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d" Oct 30 22:11:06.637481 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.637312 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0955b107-e6d3-495d-9ade-67a6f9ada6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22e8244-2qh4d\" (UID: \"0955b107-e6d3-495d-9ade-67a6f9ada6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d" Oct 30 22:11:06.637481 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.637365 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0955b107-e6d3-495d-9ade-67a6f9ada6bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22e8244-2qh4d\" (UID: \"0955b107-e6d3-495d-9ade-67a6f9ada6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d" Oct 30 22:11:06.637983 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.637954 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0955b107-e6d3-495d-9ade-67a6f9ada6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22e8244-2qh4d\" (UID: \"0955b107-e6d3-495d-9ade-67a6f9ada6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d" Oct 30 22:11:06.639992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.639961 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0955b107-e6d3-495d-9ade-67a6f9ada6bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22e8244-2qh4d\" (UID: \"0955b107-e6d3-495d-9ade-67a6f9ada6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d" Oct 30 22:11:06.698324 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.698264 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56vgz\" (UniqueName: \"kubernetes.io/projected/0955b107-e6d3-495d-9ade-67a6f9ada6bb-kube-api-access-56vgz\") pod \"maas-api-revoke-sub-f22e8244-2qh4d\" (UID: \"0955b107-e6d3-495d-9ade-67a6f9ada6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d" Oct 30 22:11:06.714187 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:06.714165 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d" Oct 30 22:11:07.104917 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:11:07.104889 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0955b107_e6d3_495d_9ade_67a6f9ada6bb.slice/crio-42accf9f0da2bd515b2197d530ac4d73b11ded0934517aa31533c3f99c72af1e WatchSource:0}: Error finding container 42accf9f0da2bd515b2197d530ac4d73b11ded0934517aa31533c3f99c72af1e: Status 404 returned error can't find the container with id 42accf9f0da2bd515b2197d530ac4d73b11ded0934517aa31533c3f99c72af1e Oct 30 22:11:07.107346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:07.107322 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d"] Oct 30 22:11:07.529507 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:07.529474 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d" event={"ID":"0955b107-e6d3-495d-9ade-67a6f9ada6bb","Type":"ContainerStarted","Data":"29964145add9f9935c968107233c250f88ab7d502ac947a71081e6a8554db70d"} Oct 30 22:11:07.529896 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:07.529537 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d" event={"ID":"0955b107-e6d3-495d-9ade-67a6f9ada6bb","Type":"ContainerStarted","Data":"42accf9f0da2bd515b2197d530ac4d73b11ded0934517aa31533c3f99c72af1e"} Oct 30 22:11:07.650671 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:07.650609 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d" podStartSLOduration=1.650596405 podStartE2EDuration="1.650596405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:07.644598651 +0000 UTC m=+1600.745030384" watchObservedRunningTime="2026-10-30 22:11:07.650596405 +0000 UTC m=+1600.751028137" Oct 30 22:11:08.076761 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:08.076723 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4"] Oct 30 22:11:08.081580 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:08.081558 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4" Oct 30 22:11:08.141259 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:08.141224 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4"] Oct 30 22:11:08.262718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:08.262686 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwm27\" (UniqueName: \"kubernetes.io/projected/c05860ee-b236-4dfa-bcb1-a38795a2c140-kube-api-access-fwm27\") pod \"maas-api-revoke-sub-8bc3318a-tmvb4\" (UID: \"c05860ee-b236-4dfa-bcb1-a38795a2c140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4" Oct 30 22:11:08.262887 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:08.262743 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05860ee-b236-4dfa-bcb1-a38795a2c140-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bc3318a-tmvb4\" (UID: \"c05860ee-b236-4dfa-bcb1-a38795a2c140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4" Oct 30 22:11:08.263010 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:08.262996 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05860ee-b236-4dfa-bcb1-a38795a2c140-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bc3318a-tmvb4\" (UID: \"c05860ee-b236-4dfa-bcb1-a38795a2c140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4" Oct 30 22:11:08.364789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:08.364718 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05860ee-b236-4dfa-bcb1-a38795a2c140-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bc3318a-tmvb4\" (UID: \"c05860ee-b236-4dfa-bcb1-a38795a2c140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4" Oct 30 22:11:08.364789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:08.364760 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwm27\" (UniqueName: \"kubernetes.io/projected/c05860ee-b236-4dfa-bcb1-a38795a2c140-kube-api-access-fwm27\") pod \"maas-api-revoke-sub-8bc3318a-tmvb4\" (UID: \"c05860ee-b236-4dfa-bcb1-a38795a2c140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4" Oct 30 22:11:08.365015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:08.364796 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05860ee-b236-4dfa-bcb1-a38795a2c140-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bc3318a-tmvb4\" (UID: \"c05860ee-b236-4dfa-bcb1-a38795a2c140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4" Oct 30 22:11:08.365530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:08.365461 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05860ee-b236-4dfa-bcb1-a38795a2c140-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bc3318a-tmvb4\" (UID: \"c05860ee-b236-4dfa-bcb1-a38795a2c140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4" Oct 30 22:11:08.367307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:08.367281 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05860ee-b236-4dfa-bcb1-a38795a2c140-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bc3318a-tmvb4\" (UID: \"c05860ee-b236-4dfa-bcb1-a38795a2c140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4" Oct 30 22:11:08.425967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:08.425937 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwm27\" (UniqueName: \"kubernetes.io/projected/c05860ee-b236-4dfa-bcb1-a38795a2c140-kube-api-access-fwm27\") pod \"maas-api-revoke-sub-8bc3318a-tmvb4\" (UID: \"c05860ee-b236-4dfa-bcb1-a38795a2c140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4" Oct 30 22:11:08.542508 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:08.542475 4335 generic.go:356] "Generic (PLEG): container finished" podID="0955b107-e6d3-495d-9ade-67a6f9ada6bb" containerID="29964145add9f9935c968107233c250f88ab7d502ac947a71081e6a8554db70d" exitCode=0 Oct 30 22:11:08.542904 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:08.542556 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d" event={"ID":"0955b107-e6d3-495d-9ade-67a6f9ada6bb","Type":"ContainerDied","Data":"29964145add9f9935c968107233c250f88ab7d502ac947a71081e6a8554db70d"} Oct 30 22:11:08.697562 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:08.697493 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4" Oct 30 22:11:09.087544 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:11:09.087517 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05860ee_b236_4dfa_bcb1_a38795a2c140.slice/crio-995bc09d125d8b8fed829e4cf67d2ec31a7d0846efa9d641859de0a01b557dcf WatchSource:0}: Error finding container 995bc09d125d8b8fed829e4cf67d2ec31a7d0846efa9d641859de0a01b557dcf: Status 404 returned error can't find the container with id 995bc09d125d8b8fed829e4cf67d2ec31a7d0846efa9d641859de0a01b557dcf Oct 30 22:11:09.090001 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:09.089975 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4"] Oct 30 22:11:09.554675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:09.554633 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4" event={"ID":"c05860ee-b236-4dfa-bcb1-a38795a2c140","Type":"ContainerStarted","Data":"77d19e00002b10042e3039793ae59e2a93e8a6778fdb68b4a7cbfb2a07e8d254"} Oct 30 22:11:09.555072 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:09.554692 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4" event={"ID":"c05860ee-b236-4dfa-bcb1-a38795a2c140","Type":"ContainerStarted","Data":"995bc09d125d8b8fed829e4cf67d2ec31a7d0846efa9d641859de0a01b557dcf"} Oct 30 22:11:09.673908 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:09.673847 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4" podStartSLOduration=1.6738294200000001 podStartE2EDuration="1.67382942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:09.669045301 +0000 UTC m=+1602.769477033" watchObservedRunningTime="2026-10-30 22:11:09.67382942 +0000 UTC m=+1602.774261152" Oct 30 22:11:09.750781 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:09.750756 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d" Oct 30 22:11:09.888094 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:09.888058 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0955b107-e6d3-495d-9ade-67a6f9ada6bb-kube-api-access-56vgz\" (UniqueName: \"kubernetes.io/projected/0955b107-e6d3-495d-9ade-67a6f9ada6bb-kube-api-access-56vgz\") pod \"0955b107-e6d3-495d-9ade-67a6f9ada6bb\" (UID: \"0955b107-e6d3-495d-9ade-67a6f9ada6bb\") " Oct 30 22:11:09.888438 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:09.888412 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0955b107-e6d3-495d-9ade-67a6f9ada6bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0955b107-e6d3-495d-9ade-67a6f9ada6bb-maas-api-openshift-ca\") pod \"0955b107-e6d3-495d-9ade-67a6f9ada6bb\" (UID: \"0955b107-e6d3-495d-9ade-67a6f9ada6bb\") " Oct 30 22:11:09.889167 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:09.889144 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0955b107-e6d3-495d-9ade-67a6f9ada6bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0955b107-e6d3-495d-9ade-67a6f9ada6bb-maas-api-xks-ca\") pod \"0955b107-e6d3-495d-9ade-67a6f9ada6bb\" (UID: \"0955b107-e6d3-495d-9ade-67a6f9ada6bb\") " Oct 30 22:11:09.890302 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:09.888938 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0955b107-e6d3-495d-9ade-67a6f9ada6bb-maas-api-openshift-ca" pod "0955b107-e6d3-495d-9ade-67a6f9ada6bb" (UID: "0955b107-e6d3-495d-9ade-67a6f9ada6bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:09.890599 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:09.890578 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0955b107-e6d3-495d-9ade-67a6f9ada6bb-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:09.893407 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:09.893371 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0955b107-e6d3-495d-9ade-67a6f9ada6bb-maas-api-xks-ca" pod "0955b107-e6d3-495d-9ade-67a6f9ada6bb" (UID: "0955b107-e6d3-495d-9ade-67a6f9ada6bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:09.893663 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:09.893632 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0955b107-e6d3-495d-9ade-67a6f9ada6bb-kube-api-access-56vgz" pod "0955b107-e6d3-495d-9ade-67a6f9ada6bb" (UID: "0955b107-e6d3-495d-9ade-67a6f9ada6bb"). InnerVolumeSpecName "kube-api-access-56vgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:09.992000 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:09.991976 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0955b107-e6d3-495d-9ade-67a6f9ada6bb-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:09.992106 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:09.992006 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56vgz\" (UniqueName: \"kubernetes.io/projected/0955b107-e6d3-495d-9ade-67a6f9ada6bb-kube-api-access-56vgz\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:10.565519 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:10.565489 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d" Oct 30 22:11:10.565519 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:10.565510 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d" event={"ID":"0955b107-e6d3-495d-9ade-67a6f9ada6bb","Type":"ContainerDied","Data":"42accf9f0da2bd515b2197d530ac4d73b11ded0934517aa31533c3f99c72af1e"} Oct 30 22:11:10.566076 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:10.565555 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42accf9f0da2bd515b2197d530ac4d73b11ded0934517aa31533c3f99c72af1e" Oct 30 22:11:10.567431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:10.567403 4335 generic.go:356] "Generic (PLEG): container finished" podID="c05860ee-b236-4dfa-bcb1-a38795a2c140" containerID="77d19e00002b10042e3039793ae59e2a93e8a6778fdb68b4a7cbfb2a07e8d254" exitCode=0 Oct 30 22:11:10.567559 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:10.567485 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4" event={"ID":"c05860ee-b236-4dfa-bcb1-a38795a2c140","Type":"ContainerDied","Data":"77d19e00002b10042e3039793ae59e2a93e8a6778fdb68b4a7cbfb2a07e8d254"} Oct 30 22:11:11.751804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:11.751779 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4" Oct 30 22:11:11.815721 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:11.815694 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c05860ee-b236-4dfa-bcb1-a38795a2c140-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05860ee-b236-4dfa-bcb1-a38795a2c140-maas-api-openshift-ca\") pod \"c05860ee-b236-4dfa-bcb1-a38795a2c140\" (UID: \"c05860ee-b236-4dfa-bcb1-a38795a2c140\") " Oct 30 22:11:11.815873 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:11.815864 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c05860ee-b236-4dfa-bcb1-a38795a2c140-kube-api-access-fwm27\" (UniqueName: \"kubernetes.io/projected/c05860ee-b236-4dfa-bcb1-a38795a2c140-kube-api-access-fwm27\") pod \"c05860ee-b236-4dfa-bcb1-a38795a2c140\" (UID: \"c05860ee-b236-4dfa-bcb1-a38795a2c140\") " Oct 30 22:11:11.815915 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:11.815897 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c05860ee-b236-4dfa-bcb1-a38795a2c140-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05860ee-b236-4dfa-bcb1-a38795a2c140-maas-api-xks-ca\") pod \"c05860ee-b236-4dfa-bcb1-a38795a2c140\" (UID: \"c05860ee-b236-4dfa-bcb1-a38795a2c140\") " Oct 30 22:11:11.816037 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:11.816017 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05860ee-b236-4dfa-bcb1-a38795a2c140-maas-api-openshift-ca" pod "c05860ee-b236-4dfa-bcb1-a38795a2c140" (UID: "c05860ee-b236-4dfa-bcb1-a38795a2c140"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:11.816380 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:11.816352 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05860ee-b236-4dfa-bcb1-a38795a2c140-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:11.818082 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:11.818057 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c05860ee-b236-4dfa-bcb1-a38795a2c140-maas-api-xks-ca" pod "c05860ee-b236-4dfa-bcb1-a38795a2c140" (UID: "c05860ee-b236-4dfa-bcb1-a38795a2c140"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:11.818204 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:11.818188 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05860ee-b236-4dfa-bcb1-a38795a2c140-kube-api-access-fwm27" pod "c05860ee-b236-4dfa-bcb1-a38795a2c140" (UID: "c05860ee-b236-4dfa-bcb1-a38795a2c140"). InnerVolumeSpecName "kube-api-access-fwm27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:11.917782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:11.917708 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwm27\" (UniqueName: \"kubernetes.io/projected/c05860ee-b236-4dfa-bcb1-a38795a2c140-kube-api-access-fwm27\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:11.917782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:11.917738 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05860ee-b236-4dfa-bcb1-a38795a2c140-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:12.588579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:12.588543 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4" Oct 30 22:11:12.588797 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:12.588601 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4" event={"ID":"c05860ee-b236-4dfa-bcb1-a38795a2c140","Type":"ContainerDied","Data":"995bc09d125d8b8fed829e4cf67d2ec31a7d0846efa9d641859de0a01b557dcf"} Oct 30 22:11:12.588797 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:12.588700 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="995bc09d125d8b8fed829e4cf67d2ec31a7d0846efa9d641859de0a01b557dcf" Oct 30 22:11:21.981317 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:21.981273 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx"] Oct 30 22:11:21.982993 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:21.982694 4335 cpu_manager.go:414] "containerMap: removing container" podUID="0955b107-e6d3-495d-9ade-67a6f9ada6bb" containerName="revoke-keys" Oct 30 22:11:21.982993 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:21.982720 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0955b107-e6d3-495d-9ade-67a6f9ada6bb" containerName="revoke-keys" Oct 30 22:11:21.982993 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:21.982793 4335 cpu_manager.go:414] "containerMap: removing container" podUID="c05860ee-b236-4dfa-bcb1-a38795a2c140" containerName="revoke-keys" Oct 30 22:11:21.982993 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:21.982802 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c05860ee-b236-4dfa-bcb1-a38795a2c140" containerName="revoke-keys" Oct 30 22:11:21.983364 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:21.983165 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="0955b107-e6d3-495d-9ade-67a6f9ada6bb" containerName="revoke-keys" Oct 30 22:11:21.983364 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:21.983195 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="c05860ee-b236-4dfa-bcb1-a38795a2c140" containerName="revoke-keys" Oct 30 22:11:21.996955 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:21.996931 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.058112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.058080 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-aa8c82d5-openshift-default-dockercfg-r2zpw\"" Oct 30 22:11:22.100718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.077602 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-data\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.101052 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.101027 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-credential-socket\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.121636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.109735 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-token\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.130714 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.121943 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-workload-socket\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.203963 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.203918 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-workload-certs\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.204177 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.204014 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ed8581-628f-42ce-8ad0-8899cce519f2-istiod-ca-cert\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.204177 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.204159 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl5zf\" (UniqueName: \"kubernetes.io/projected/52ed8581-628f-42ce-8ad0-8899cce519f2-kube-api-access-gl5zf\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.204324 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.204283 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-podinfo\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.204388 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.204363 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-envoy\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.211726 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.211700 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx"] Oct 30 22:11:22.312290 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.312235 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-data\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.312522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.312333 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-credential-socket\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.312522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.312396 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-token\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.312662 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.312516 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-workload-socket\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.312662 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.312578 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-workload-certs\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.312791 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.312659 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ed8581-628f-42ce-8ad0-8899cce519f2-istiod-ca-cert\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.312852 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.312784 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl5zf\" (UniqueName: \"kubernetes.io/projected/52ed8581-628f-42ce-8ad0-8899cce519f2-kube-api-access-gl5zf\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.312909 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.312876 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-podinfo\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.312999 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.312979 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-envoy\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.314968 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.314933 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-data\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.315198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.315172 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-workload-socket\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.315302 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.315227 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-workload-certs\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.315842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.315818 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-credential-socket\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.315994 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.315918 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ed8581-628f-42ce-8ad0-8899cce519f2-istiod-ca-cert\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.317760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.317719 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-envoy\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.339404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.338518 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-podinfo\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.381095 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.379837 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl5zf\" (UniqueName: \"kubernetes.io/projected/52ed8581-628f-42ce-8ad0-8899cce519f2-kube-api-access-gl5zf\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.381095 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.380380 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-token\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:22.647002 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:22.646848 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:23.486877 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:11:23.484814 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ed8581_628f_42ce_8ad0_8899cce519f2.slice/crio-fb635dde3caae9992fd0e97e939a8e08e1a4d012d234cdab4c1b2ad269b57286 WatchSource:0}: Error finding container fb635dde3caae9992fd0e97e939a8e08e1a4d012d234cdab4c1b2ad269b57286: Status 404 returned error can't find the container with id fb635dde3caae9992fd0e97e939a8e08e1a4d012d234cdab4c1b2ad269b57286 Oct 30 22:11:23.495707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:23.487743 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:11:23.495707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:23.487843 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:11:23.495707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:23.487896 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:11:23.495707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:23.489012 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx"] Oct 30 22:11:23.758690 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:23.757649 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" event={"ID":"52ed8581-628f-42ce-8ad0-8899cce519f2","Type":"ContainerStarted","Data":"931591b0ddb2a6689f4d5772133a0a60027e4ac44619f0bafad46ad29c55ef4b"} Oct 30 22:11:23.758690 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:23.757711 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" event={"ID":"52ed8581-628f-42ce-8ad0-8899cce519f2","Type":"ContainerStarted","Data":"fb635dde3caae9992fd0e97e939a8e08e1a4d012d234cdab4c1b2ad269b57286"} Oct 30 22:11:23.888125 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:23.888054 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" podStartSLOduration=2.888036546 podStartE2EDuration="2.888036546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:23.877921838 +0000 UTC m=+1616.978353593" watchObservedRunningTime="2026-10-30 22:11:23.888036546 +0000 UTC m=+1616.988468281" Oct 30 22:11:24.675285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:24.665947 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:24.675285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:24.674910 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:24.830646 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:24.829727 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:24.848750 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:24.847001 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:11:25.350645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:25.349718 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:11:25.350645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:25.349784 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" podUID="4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:25.496642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:25.495966 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:11:25.496642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:25.496028 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" podUID="87f6d9bc-feca-46a3-925f-e3402a1d895c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:25.519814 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:11:25.519760 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:11:27.846990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:27.846954 4335 scope.go:122] "RemoveContainer" containerID="76b558689b717d9340ce40d63ff336c71164d03aa12752fa43c4834fbe74cfda" Oct 30 22:11:27.861501 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:27.861479 4335 scope.go:122] "RemoveContainer" containerID="04c3444def8bce47b441ced27af67b83479e52253f6f5d150e70cb92fecd312c" Oct 30 22:11:27.879478 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:27.879456 4335 scope.go:122] "RemoveContainer" containerID="f1d78ece05784a6800a18f5ae961f6488558a6505dcc937984213ad6595f79ee" Oct 30 22:11:27.893095 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:27.893072 4335 scope.go:122] "RemoveContainer" containerID="2cad413f5c27cf55e67d179d9516b4b91e5c8e94cf9f4aa899bbe59970326c33" Oct 30 22:11:31.482405 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.482370 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m"] Oct 30 22:11:31.491350 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.491316 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m" Oct 30 22:11:31.506704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.506674 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hkd7\" (UniqueName: \"kubernetes.io/projected/4dedd858-dcf5-49e5-83c6-abfbd08e8224-kube-api-access-4hkd7\") pod \"payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m\" (UID: \"4dedd858-dcf5-49e5-83c6-abfbd08e8224\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m" Oct 30 22:11:31.506932 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.506909 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dedd858-dcf5-49e5-83c6-abfbd08e8224-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m\" (UID: \"4dedd858-dcf5-49e5-83c6-abfbd08e8224\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m" Oct 30 22:11:31.527373 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.527345 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m"] Oct 30 22:11:31.543608 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.543574 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-aa8c82d5\"" Oct 30 22:11:31.602685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.602640 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj"] Oct 30 22:11:31.608442 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.608408 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dedd858-dcf5-49e5-83c6-abfbd08e8224-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m\" (UID: \"4dedd858-dcf5-49e5-83c6-abfbd08e8224\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m" Oct 30 22:11:31.608683 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.608578 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hkd7\" (UniqueName: \"kubernetes.io/projected/4dedd858-dcf5-49e5-83c6-abfbd08e8224-kube-api-access-4hkd7\") pod \"payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m\" (UID: \"4dedd858-dcf5-49e5-83c6-abfbd08e8224\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m" Oct 30 22:11:31.608773 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.608693 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj" Oct 30 22:11:31.609338 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.609314 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dedd858-dcf5-49e5-83c6-abfbd08e8224-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m\" (UID: \"4dedd858-dcf5-49e5-83c6-abfbd08e8224\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m" Oct 30 22:11:31.662870 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.662839 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-aa8c82d5-dockercfg-9bglr\"" Oct 30 22:11:31.671430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.671399 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj"] Oct 30 22:11:31.671737 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.671713 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hkd7\" (UniqueName: \"kubernetes.io/projected/4dedd858-dcf5-49e5-83c6-abfbd08e8224-kube-api-access-4hkd7\") pod \"payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m\" (UID: \"4dedd858-dcf5-49e5-83c6-abfbd08e8224\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m" Oct 30 22:11:31.710099 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.710065 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f996c1c2-bd90-4b41-99b3-08f969d9aba5-plugins-config-volume\") pod \"payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj\" (UID: \"f996c1c2-bd90-4b41-99b3-08f969d9aba5\") " pod="openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj" Oct 30 22:11:31.710348 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.710324 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgrqz\" (UniqueName: \"kubernetes.io/projected/f996c1c2-bd90-4b41-99b3-08f969d9aba5-kube-api-access-vgrqz\") pod \"payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj\" (UID: \"f996c1c2-bd90-4b41-99b3-08f969d9aba5\") " pod="openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj" Oct 30 22:11:31.818641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.811074 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m" Oct 30 22:11:31.818641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.812290 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgrqz\" (UniqueName: \"kubernetes.io/projected/f996c1c2-bd90-4b41-99b3-08f969d9aba5-kube-api-access-vgrqz\") pod \"payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj\" (UID: \"f996c1c2-bd90-4b41-99b3-08f969d9aba5\") " pod="openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj" Oct 30 22:11:31.818641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.812539 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f996c1c2-bd90-4b41-99b3-08f969d9aba5-plugins-config-volume\") pod \"payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj\" (UID: \"f996c1c2-bd90-4b41-99b3-08f969d9aba5\") " pod="openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj" Oct 30 22:11:31.818641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.813776 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f996c1c2-bd90-4b41-99b3-08f969d9aba5-plugins-config-volume\") pod \"payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj\" (UID: \"f996c1c2-bd90-4b41-99b3-08f969d9aba5\") " pod="openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj" Oct 30 22:11:31.906708 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.887245 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgrqz\" (UniqueName: \"kubernetes.io/projected/f996c1c2-bd90-4b41-99b3-08f969d9aba5-kube-api-access-vgrqz\") pod \"payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj\" (UID: \"f996c1c2-bd90-4b41-99b3-08f969d9aba5\") " pod="openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj" Oct 30 22:11:31.931875 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:31.931846 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj" Oct 30 22:11:32.356169 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:32.356102 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m"] Oct 30 22:11:32.394943 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:11:32.394915 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dedd858_dcf5_49e5_83c6_abfbd08e8224.slice/crio-d53e3a8bba0072ecfc87632251dd39453c9aee78894aa380c9fc16b11ba22305 WatchSource:0}: Error finding container d53e3a8bba0072ecfc87632251dd39453c9aee78894aa380c9fc16b11ba22305: Status 404 returned error can't find the container with id d53e3a8bba0072ecfc87632251dd39453c9aee78894aa380c9fc16b11ba22305 Oct 30 22:11:32.411474 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:32.411431 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj"] Oct 30 22:11:32.412525 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:11:32.412497 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf996c1c2_bd90_4b41_99b3_08f969d9aba5.slice/crio-173273a5b99ceda1a98fd1583d2a80a339d1813049c930bd3a54f9ff089df7c7 WatchSource:0}: Error finding container 173273a5b99ceda1a98fd1583d2a80a339d1813049c930bd3a54f9ff089df7c7: Status 404 returned error can't find the container with id 173273a5b99ceda1a98fd1583d2a80a339d1813049c930bd3a54f9ff089df7c7 Oct 30 22:11:33.006557 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:33.006520 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj" event={"ID":"f996c1c2-bd90-4b41-99b3-08f969d9aba5","Type":"ContainerStarted","Data":"c8b31ae1ada9855d1b1609f5583e7aa7b8765bcb02d5c1d53f9b65cccc227955"} Oct 30 22:11:33.007039 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:33.006571 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj" event={"ID":"f996c1c2-bd90-4b41-99b3-08f969d9aba5","Type":"ContainerStarted","Data":"173273a5b99ceda1a98fd1583d2a80a339d1813049c930bd3a54f9ff089df7c7"} Oct 30 22:11:33.007039 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:33.006648 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj" Oct 30 22:11:33.008690 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:33.008658 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m" event={"ID":"4dedd858-dcf5-49e5-83c6-abfbd08e8224","Type":"ContainerStarted","Data":"5261d3c9fde1aecf5a437af6e184786c41d86500f0c3d796eb75ef7c43dde878"} Oct 30 22:11:33.008829 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:33.008710 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m" event={"ID":"4dedd858-dcf5-49e5-83c6-abfbd08e8224","Type":"ContainerStarted","Data":"d53e3a8bba0072ecfc87632251dd39453c9aee78894aa380c9fc16b11ba22305"} Oct 30 22:11:33.008829 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:33.008737 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m" Oct 30 22:11:33.126654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:33.126570 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj" podStartSLOduration=2.126557695 podStartE2EDuration="2.126557695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:33.122192754 +0000 UTC m=+1626.222624487" watchObservedRunningTime="2026-10-30 22:11:33.126557695 +0000 UTC m=+1626.226989426" Oct 30 22:11:33.242528 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:33.242463 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m" podStartSLOduration=2.24244838 podStartE2EDuration="2.24244838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:33.236794947 +0000 UTC m=+1626.337226679" watchObservedRunningTime="2026-10-30 22:11:33.24244838 +0000 UTC m=+1626.342880111" Oct 30 22:11:41.323164 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.323134 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km"] Oct 30 22:11:41.329354 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.329333 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.381423 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.381396 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 22:11:41.381560 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.381397 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 22:11:41.381560 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.381404 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 22:11:41.381560 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.381405 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mjfxd\"" Oct 30 22:11:41.438762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.438738 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km"] Oct 30 22:11:41.457124 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.457101 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.457267 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.457170 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.457340 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.457321 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.457378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.457353 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4sqd\" (UniqueName: \"kubernetes.io/projected/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-kube-api-access-x4sqd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.457417 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.457383 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.457472 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.457456 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.457544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.457530 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.559455 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.559429 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.559595 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.559464 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4sqd\" (UniqueName: \"kubernetes.io/projected/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-kube-api-access-x4sqd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.559595 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.559487 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.559595 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.559505 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.559840 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.559694 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.559890 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.559877 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.560000 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.559978 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.560507 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.560476 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.562214 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.562191 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.562304 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.562224 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.562304 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.562197 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.562539 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.562523 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.620180 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.620128 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4sqd\" (UniqueName: \"kubernetes.io/projected/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-kube-api-access-x4sqd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.620267 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.620242 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-299km\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:41.643987 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:41.643966 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:42.079856 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:11:42.079822 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f0b281_9719_4ec5_a5d7_2a6a4ad0b5cc.slice/crio-2a7d813f453ada802f2c59cea8b322d9e6df0ea7c87d5bddeddc6975448698db WatchSource:0}: Error finding container 2a7d813f453ada802f2c59cea8b322d9e6df0ea7c87d5bddeddc6975448698db: Status 404 returned error can't find the container with id 2a7d813f453ada802f2c59cea8b322d9e6df0ea7c87d5bddeddc6975448698db Oct 30 22:11:42.081182 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:42.081157 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km"] Oct 30 22:11:42.101356 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:42.101325 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" event={"ID":"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc","Type":"ContainerStarted","Data":"2a7d813f453ada802f2c59cea8b322d9e6df0ea7c87d5bddeddc6975448698db"} Oct 30 22:11:44.021707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:44.021673 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m" Oct 30 22:11:44.026634 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:44.024734 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj" Oct 30 22:11:46.218154 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:46.218116 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:11:46.218573 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:46.218199 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:11:47.156268 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:47.156230 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" event={"ID":"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc","Type":"ContainerStarted","Data":"c434891978a4d097928534d752b6bf1d865a0f21e76963ef3388203cac57b824"} Oct 30 22:11:47.156447 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:47.156290 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:47.158345 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:47.158319 4335 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-299km container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:11:47.158499 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:47.158387 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" podUID="c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:11:47.377321 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:47.377251 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" podStartSLOduration=2.242400361 podStartE2EDuration="6.377234847s" totalImagesPullingTime="4.134834486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:47.335728354 +0000 UTC m=+1640.436160092" watchObservedRunningTime="2026-10-30 22:11:47.377234847 +0000 UTC m=+1640.477666585" Oct 30 22:11:48.175717 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.175322 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:11:48.685903 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.685871 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss"] Oct 30 22:11:48.691828 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.691804 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.776950 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.776912 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss"] Oct 30 22:11:48.790230 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.790194 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.790403 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.790258 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.790403 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.790288 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.790403 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.790335 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.790403 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.790382 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtwkf\" (UniqueName: \"kubernetes.io/projected/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-kube-api-access-rtwkf\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.790691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.790432 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.790691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.790515 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.790822 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.790688 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.790822 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.790792 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.836725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.836694 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v5mg8\"" Oct 30 22:11:48.892021 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.891993 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.892125 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.892077 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.892167 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.892153 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.892202 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.892184 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.892240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.892200 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.892240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.892235 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.892324 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.892276 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtwkf\" (UniqueName: \"kubernetes.io/projected/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-kube-api-access-rtwkf\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.892380 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.892320 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.892431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.892389 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.892499 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.892475 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.892826 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.892809 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.892958 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.892937 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.892998 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.892963 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.893033 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.893023 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.895183 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.895163 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.895248 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.895223 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.953691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.953595 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:48.953828 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:48.953804 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtwkf\" (UniqueName: \"kubernetes.io/projected/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-kube-api-access-rtwkf\") pod \"maas-default-gateway-openshift-default-6cc458d887-n6mss\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:49.007758 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:49.007735 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:49.459697 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:11:49.459666 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b5d5ce_678d_4e48_87f7_5d6b2524ceae.slice/crio-e5d259b162f8b95239bba563bf5e5782de7902bd0a7f46aa6191cb8f34794436 WatchSource:0}: Error finding container e5d259b162f8b95239bba563bf5e5782de7902bd0a7f46aa6191cb8f34794436: Status 404 returned error can't find the container with id e5d259b162f8b95239bba563bf5e5782de7902bd0a7f46aa6191cb8f34794436 Oct 30 22:11:49.461842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:49.461817 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss"] Oct 30 22:11:49.462540 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:49.462499 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:11:49.462656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:49.462592 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:11:49.462656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:49.462648 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:11:50.222507 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:50.222469 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" event={"ID":"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae","Type":"ContainerStarted","Data":"2a21a39e8d02929f95dae2ccb10b46f41b54a8a8275d2d52918263a666839e16"} Oct 30 22:11:50.228145 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:50.227687 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" event={"ID":"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae","Type":"ContainerStarted","Data":"e5d259b162f8b95239bba563bf5e5782de7902bd0a7f46aa6191cb8f34794436"} Oct 30 22:11:50.442835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:50.442761 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" podStartSLOduration=2.442740333 podStartE2EDuration="2.442740333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:50.422068011 +0000 UTC m=+1643.522499745" watchObservedRunningTime="2026-10-30 22:11:50.442740333 +0000 UTC m=+1643.543172066" Oct 30 22:11:51.019926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:51.019894 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:51.021835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:51.021596 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n6mss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 22:11:51.021835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:51.021695 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" podUID="e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 22:11:52.015639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:52.014939 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n6mss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 22:11:52.015639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:52.015015 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" podUID="e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 22:11:53.012953 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:53.012918 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n6mss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 22:11:53.013363 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:53.013331 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" podUID="e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 22:11:54.008967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:54.008929 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n6mss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 22:11:54.009554 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:54.009522 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" podUID="e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 22:11:55.014471 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:55.013975 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n6mss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 22:11:55.014471 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:55.014049 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" podUID="e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 22:11:55.346192 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:11:55.346160 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:11:56.009085 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:56.009050 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n6mss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 22:11:56.009286 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:56.009142 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" podUID="e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 22:11:57.008403 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:57.008367 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n6mss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 22:11:57.008831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:57.008436 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" podUID="e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 22:11:58.015629 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:58.015586 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:58.016169 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:58.016146 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:11:58.016993 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:11:58.016974 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:12:03.064345 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:03.064316 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd"] Oct 30 22:12:03.105175 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:03.105145 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b84145f-4xqvd"] Oct 30 22:12:04.641397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:04.641360 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:12:05.059718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:05.059682 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba146f29-2902-499d-9c92-93b1cb3beeb6" path="/var/lib/kubelet/pods/ba146f29-2902-499d-9c92-93b1cb3beeb6/volumes" Oct 30 22:12:06.767642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:06.765879 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 22:12:06.767642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:06.767081 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x" podUID="1023f325-f4c6-41ae-86f5-d0796101a380" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:06.790745 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:06.790701 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:06.826648 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:06.824707 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:06.885106 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:06.884711 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:06.958643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:06.948734 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Oct 30 22:12:07.009031 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:07.008988 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:07.116277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:07.116218 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:07.328696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:07.328640 4335 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/health\": context deadline exceeded" Oct 30 22:12:07.433699 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:07.420724 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:07.536688 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:07.532758 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:16.740218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:16.740153 4335 generic.go:356] "Generic (PLEG): container finished" podID="27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" containerID="c379493889432c301d6e169ec821fe5a8d11569eff5d73932e5176d44ad50c51" exitCode=0 Oct 30 22:12:16.740591 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:16.740236 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" event={"ID":"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0","Type":"ContainerDied","Data":"c379493889432c301d6e169ec821fe5a8d11569eff5d73932e5176d44ad50c51"} Oct 30 22:12:16.740692 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:16.740665 4335 scope.go:122] "RemoveContainer" containerID="c379493889432c301d6e169ec821fe5a8d11569eff5d73932e5176d44ad50c51" Oct 30 22:12:16.742351 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:16.742322 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:12:16.742456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:16.742409 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:12:16.742522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:16.742465 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:12:17.694486 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:17.694458 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:12:17.761289 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:17.761249 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" event={"ID":"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0","Type":"ContainerStarted","Data":"5491bab0f2a7b3c4bf92a55ecf21b6ba8e5962c8f74931d15c89ff799cc2062e"} Oct 30 22:12:17.767661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:17.764948 4335 generic.go:356] "Generic (PLEG): container finished" podID="576a247d-5dc2-43f5-9f67-4b572ea9a072" containerID="ca898a0d633cfa4c00ee3552245837019b7e807ce4742781dbc44c08d7df89a6" exitCode=0 Oct 30 22:12:17.767661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:17.765169 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" event={"ID":"576a247d-5dc2-43f5-9f67-4b572ea9a072","Type":"ContainerDied","Data":"ca898a0d633cfa4c00ee3552245837019b7e807ce4742781dbc44c08d7df89a6"} Oct 30 22:12:17.767661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:17.765935 4335 scope.go:122] "RemoveContainer" containerID="ca898a0d633cfa4c00ee3552245837019b7e807ce4742781dbc44c08d7df89a6" Oct 30 22:12:17.769146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:17.769063 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:12:17.769298 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:17.769248 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:12:17.769375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:17.769354 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:12:18.275670 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:18.272714 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:12:18.283443 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:18.283391 4335 patch_prober.go:29] interesting pod/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 22:12:18.283700 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:18.283452 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" podUID="27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 22:12:18.786048 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:18.786019 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" event={"ID":"576a247d-5dc2-43f5-9f67-4b572ea9a072","Type":"ContainerStarted","Data":"684c28e19754bc3e20270bed0c683050a9f0f2621a501888c2f5e1c126731a77"} Oct 30 22:12:19.124650 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:19.124423 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:12:19.124650 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:19.124556 4335 patch_prober.go:29] interesting pod/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 22:12:19.124650 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:19.124600 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" podUID="576a247d-5dc2-43f5-9f67-4b572ea9a072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 22:12:19.273069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:19.273036 4335 patch_prober.go:29] interesting pod/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 22:12:19.273296 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:19.273276 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" podUID="27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 22:12:20.134654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:20.133985 4335 patch_prober.go:29] interesting pod/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 22:12:20.134654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:20.134045 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" podUID="576a247d-5dc2-43f5-9f67-4b572ea9a072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 22:12:20.273048 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:20.273014 4335 patch_prober.go:29] interesting pod/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 22:12:20.273198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:20.273067 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" podUID="27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 22:12:21.125642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:21.124015 4335 patch_prober.go:29] interesting pod/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 22:12:21.125642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:21.124077 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" podUID="576a247d-5dc2-43f5-9f67-4b572ea9a072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 22:12:21.290303 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:21.289018 4335 patch_prober.go:29] interesting pod/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 22:12:21.290303 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:21.289078 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" podUID="27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 22:12:22.127639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:22.127044 4335 patch_prober.go:29] interesting pod/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 22:12:22.127639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:22.127136 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" podUID="576a247d-5dc2-43f5-9f67-4b572ea9a072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 22:12:22.272084 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:22.272041 4335 patch_prober.go:29] interesting pod/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 22:12:22.272254 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:22.272106 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" podUID="27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 22:12:23.144762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:23.136733 4335 patch_prober.go:29] interesting pod/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 22:12:23.144762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:23.136836 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" podUID="576a247d-5dc2-43f5-9f67-4b572ea9a072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 22:12:23.203641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:23.203449 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:12:23.272830 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:23.272797 4335 patch_prober.go:29] interesting pod/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 22:12:23.273036 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:23.272851 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" podUID="27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 22:12:24.133680 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:24.133057 4335 patch_prober.go:29] interesting pod/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 22:12:24.133680 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:24.133131 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" podUID="576a247d-5dc2-43f5-9f67-4b572ea9a072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 22:12:24.276515 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:24.276486 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:12:24.276949 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:24.276924 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:12:24.277675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:24.277658 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:12:25.128256 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:25.128224 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:12:25.129532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:25.129510 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:12:25.325427 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:12:25.325393 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:12:33.303563 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:33.303525 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn"] Oct 30 22:12:33.312224 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:33.312200 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn" Oct 30 22:12:33.368657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:33.368602 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn"] Oct 30 22:12:33.408964 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:33.408938 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:12:33.408964 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:33.408958 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:12:33.408964 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:33.408941 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:33.413245 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:33.413220 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:12:33.439576 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:33.439551 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh7x9\" (UniqueName: \"kubernetes.io/projected/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-kube-api-access-nh7x9\") pod \"maas-api-revoke-keys-e2e-migrate-92ce375b-826xn\" (UID: \"5fc02abe-0c34-43b8-9ffa-9cb96352cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn" Oct 30 22:12:33.439752 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:33.439590 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-92ce375b-826xn\" (UID: \"5fc02abe-0c34-43b8-9ffa-9cb96352cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn" Oct 30 22:12:33.439880 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:33.439847 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-92ce375b-826xn\" (UID: \"5fc02abe-0c34-43b8-9ffa-9cb96352cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn" Oct 30 22:12:33.541295 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:33.541269 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh7x9\" (UniqueName: \"kubernetes.io/projected/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-kube-api-access-nh7x9\") pod \"maas-api-revoke-keys-e2e-migrate-92ce375b-826xn\" (UID: \"5fc02abe-0c34-43b8-9ffa-9cb96352cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn" Oct 30 22:12:33.541418 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:33.541304 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-92ce375b-826xn\" (UID: \"5fc02abe-0c34-43b8-9ffa-9cb96352cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn" Oct 30 22:12:33.541418 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:33.541380 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-92ce375b-826xn\" (UID: \"5fc02abe-0c34-43b8-9ffa-9cb96352cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn" Oct 30 22:12:33.541978 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:33.541959 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-92ce375b-826xn\" (UID: \"5fc02abe-0c34-43b8-9ffa-9cb96352cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn" Oct 30 22:12:33.543929 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:33.543911 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-92ce375b-826xn\" (UID: \"5fc02abe-0c34-43b8-9ffa-9cb96352cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn" Oct 30 22:12:33.602693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:33.602637 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh7x9\" (UniqueName: \"kubernetes.io/projected/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-kube-api-access-nh7x9\") pod \"maas-api-revoke-keys-e2e-migrate-92ce375b-826xn\" (UID: \"5fc02abe-0c34-43b8-9ffa-9cb96352cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn" Oct 30 22:12:33.626779 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:33.626760 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn" Oct 30 22:12:34.019921 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:12:34.019522 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc02abe_0c34_43b8_9ffa_9cb96352cc57.slice/crio-5d5c9149cfaadccc3836660f4bb8bae50070543a6f7076403548d2ab45f8163e WatchSource:0}: Error finding container 5d5c9149cfaadccc3836660f4bb8bae50070543a6f7076403548d2ab45f8163e: Status 404 returned error can't find the container with id 5d5c9149cfaadccc3836660f4bb8bae50070543a6f7076403548d2ab45f8163e Oct 30 22:12:34.023564 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.023540 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn"] Oct 30 22:12:34.040386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.040356 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn" event={"ID":"5fc02abe-0c34-43b8-9ffa-9cb96352cc57","Type":"ContainerStarted","Data":"5d5c9149cfaadccc3836660f4bb8bae50070543a6f7076403548d2ab45f8163e"} Oct 30 22:12:34.071060 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.071031 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4"] Oct 30 22:12:34.079743 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.079679 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.149975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.149940 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.150139 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.150018 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.150139 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.150057 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.150139 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.150128 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.150330 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.150310 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.150394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.150348 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.150457 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.150426 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4116062-6db3-424f-b4cc-3442ab74a3b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.150457 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.150451 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgrc4\" (UniqueName: \"kubernetes.io/projected/c4116062-6db3-424f-b4cc-3442ab74a3b2-kube-api-access-wgrc4\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.150570 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.150478 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.151006 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.150983 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4"] Oct 30 22:12:34.251727 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.251697 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.251886 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.251739 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.251886 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.251789 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4116062-6db3-424f-b4cc-3442ab74a3b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.251886 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.251816 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgrc4\" (UniqueName: \"kubernetes.io/projected/c4116062-6db3-424f-b4cc-3442ab74a3b2-kube-api-access-wgrc4\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.251886 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.251841 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.252137 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.251897 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.252137 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.251947 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.252137 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.251979 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.252137 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.252045 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.252354 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.252172 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.252499 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.252472 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.252570 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.252508 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.252671 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.252579 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4116062-6db3-424f-b4cc-3442ab74a3b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.252791 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.252769 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.255500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.255472 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.255987 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.255964 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.396706 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.396672 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgrc4\" (UniqueName: \"kubernetes.io/projected/c4116062-6db3-424f-b4cc-3442ab74a3b2-kube-api-access-wgrc4\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.396706 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.396690 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-blzr4\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.400785 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.400763 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:34.844681 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:12:34.844607 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4116062_6db3_424f_b4cc_3442ab74a3b2.slice/crio-c7a155080097f8799d7c6fa1bdec3eb44e938f6e28c5a208e5fe7b0b7004b389 WatchSource:0}: Error finding container c7a155080097f8799d7c6fa1bdec3eb44e938f6e28c5a208e5fe7b0b7004b389: Status 404 returned error can't find the container with id c7a155080097f8799d7c6fa1bdec3eb44e938f6e28c5a208e5fe7b0b7004b389 Oct 30 22:12:34.848163 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.848126 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:12:34.848267 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.848214 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:12:34.848267 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.848258 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:12:34.848900 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:34.848856 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4"] Oct 30 22:12:35.062145 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:35.062106 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn" event={"ID":"5fc02abe-0c34-43b8-9ffa-9cb96352cc57","Type":"ContainerStarted","Data":"344fe711fc609654437a2a71a769ddce1f8ad3b712e6bdb549ed6d212f2c20ad"} Oct 30 22:12:35.062300 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:35.062156 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" event={"ID":"c4116062-6db3-424f-b4cc-3442ab74a3b2","Type":"ContainerStarted","Data":"9b7a61f41dda4ce6157b4630ee8bde8f4e0ddc56161a7d3706c3d417f75f9752"} Oct 30 22:12:35.062300 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:35.062180 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" event={"ID":"c4116062-6db3-424f-b4cc-3442ab74a3b2","Type":"ContainerStarted","Data":"c7a155080097f8799d7c6fa1bdec3eb44e938f6e28c5a208e5fe7b0b7004b389"} Oct 30 22:12:35.178104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:35.178047 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn" podStartSLOduration=2.17802804 podStartE2EDuration="2.17802804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:35.168410646 +0000 UTC m=+1688.268842378" watchObservedRunningTime="2026-10-30 22:12:35.17802804 +0000 UTC m=+1688.278459774" Oct 30 22:12:35.289895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:35.289847 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" podStartSLOduration=1.289833638 podStartE2EDuration="1.289833638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:35.287074049 +0000 UTC m=+1688.387505794" watchObservedRunningTime="2026-10-30 22:12:35.289833638 +0000 UTC m=+1688.390265368" Oct 30 22:12:35.401889 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:35.401798 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:35.404240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:35.404208 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-blzr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 22:12:35.404474 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:35.404435 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" podUID="c4116062-6db3-424f-b4cc-3442ab74a3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 22:12:36.067834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:36.067794 4335 generic.go:356] "Generic (PLEG): container finished" podID="5fc02abe-0c34-43b8-9ffa-9cb96352cc57" containerID="344fe711fc609654437a2a71a769ddce1f8ad3b712e6bdb549ed6d212f2c20ad" exitCode=0 Oct 30 22:12:36.068027 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:36.067858 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn" event={"ID":"5fc02abe-0c34-43b8-9ffa-9cb96352cc57","Type":"ContainerDied","Data":"344fe711fc609654437a2a71a769ddce1f8ad3b712e6bdb549ed6d212f2c20ad"} Oct 30 22:12:36.325533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:36.325447 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt"] Oct 30 22:12:36.331375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:36.331346 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt" Oct 30 22:12:36.371997 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:36.371965 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt"] Oct 30 22:12:36.386444 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:36.386207 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4175f9-5637-46d2-a593-ca01d9b970ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt\" (UID: \"9b4175f9-5637-46d2-a593-ca01d9b970ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt" Oct 30 22:12:36.386444 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:36.386327 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbw4m\" (UniqueName: \"kubernetes.io/projected/9b4175f9-5637-46d2-a593-ca01d9b970ed-kube-api-access-fbw4m\") pod \"maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt\" (UID: \"9b4175f9-5637-46d2-a593-ca01d9b970ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt" Oct 30 22:12:36.386677 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:36.386658 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4175f9-5637-46d2-a593-ca01d9b970ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt\" (UID: \"9b4175f9-5637-46d2-a593-ca01d9b970ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt" Oct 30 22:12:36.418638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:36.403971 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-blzr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 22:12:36.418638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:36.404023 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" podUID="c4116062-6db3-424f-b4cc-3442ab74a3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 22:12:36.488139 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:36.488107 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4175f9-5637-46d2-a593-ca01d9b970ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt\" (UID: \"9b4175f9-5637-46d2-a593-ca01d9b970ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt" Oct 30 22:12:36.488310 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:36.488185 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbw4m\" (UniqueName: \"kubernetes.io/projected/9b4175f9-5637-46d2-a593-ca01d9b970ed-kube-api-access-fbw4m\") pod \"maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt\" (UID: \"9b4175f9-5637-46d2-a593-ca01d9b970ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt" Oct 30 22:12:36.488500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:36.488485 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4175f9-5637-46d2-a593-ca01d9b970ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt\" (UID: \"9b4175f9-5637-46d2-a593-ca01d9b970ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt" Oct 30 22:12:36.489137 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:36.489115 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4175f9-5637-46d2-a593-ca01d9b970ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt\" (UID: \"9b4175f9-5637-46d2-a593-ca01d9b970ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt" Oct 30 22:12:36.490919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:36.490898 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4175f9-5637-46d2-a593-ca01d9b970ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt\" (UID: \"9b4175f9-5637-46d2-a593-ca01d9b970ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt" Oct 30 22:12:36.549925 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:36.549890 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbw4m\" (UniqueName: \"kubernetes.io/projected/9b4175f9-5637-46d2-a593-ca01d9b970ed-kube-api-access-fbw4m\") pod \"maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt\" (UID: \"9b4175f9-5637-46d2-a593-ca01d9b970ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt" Oct 30 22:12:36.646707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:36.646635 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt" Oct 30 22:12:37.066596 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:12:37.066562 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4175f9_5637_46d2_a593_ca01d9b970ed.slice/crio-1d0fb0e997c016b7af1768d6f4954cfb2fcbfc593a41c6ab975f5c39a0aefd54 WatchSource:0}: Error finding container 1d0fb0e997c016b7af1768d6f4954cfb2fcbfc593a41c6ab975f5c39a0aefd54: Status 404 returned error can't find the container with id 1d0fb0e997c016b7af1768d6f4954cfb2fcbfc593a41c6ab975f5c39a0aefd54 Oct 30 22:12:37.070566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:37.070540 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt"] Oct 30 22:12:37.078677 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:37.078642 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt" event={"ID":"9b4175f9-5637-46d2-a593-ca01d9b970ed","Type":"ContainerStarted","Data":"1d0fb0e997c016b7af1768d6f4954cfb2fcbfc593a41c6ab975f5c39a0aefd54"} Oct 30 22:12:37.260067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:37.260039 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn" Oct 30 22:12:37.436639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:37.434113 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-blzr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 22:12:37.436639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:37.434170 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" podUID="c4116062-6db3-424f-b4cc-3442ab74a3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 22:12:37.436639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:37.435151 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-kube-api-access-nh7x9\" (UniqueName: \"kubernetes.io/projected/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-kube-api-access-nh7x9\") pod \"5fc02abe-0c34-43b8-9ffa-9cb96352cc57\" (UID: \"5fc02abe-0c34-43b8-9ffa-9cb96352cc57\") " Oct 30 22:12:37.436639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:37.435402 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-maas-api-xks-ca\") pod \"5fc02abe-0c34-43b8-9ffa-9cb96352cc57\" (UID: \"5fc02abe-0c34-43b8-9ffa-9cb96352cc57\") " Oct 30 22:12:37.436639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:37.435472 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-maas-api-openshift-ca\") pod \"5fc02abe-0c34-43b8-9ffa-9cb96352cc57\" (UID: \"5fc02abe-0c34-43b8-9ffa-9cb96352cc57\") " Oct 30 22:12:37.462474 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:37.462439 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-maas-api-openshift-ca" pod "5fc02abe-0c34-43b8-9ffa-9cb96352cc57" (UID: "5fc02abe-0c34-43b8-9ffa-9cb96352cc57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:37.465217 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:37.465171 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-maas-api-xks-ca" pod "5fc02abe-0c34-43b8-9ffa-9cb96352cc57" (UID: "5fc02abe-0c34-43b8-9ffa-9cb96352cc57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:37.475774 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:37.474932 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-kube-api-access-nh7x9" pod "5fc02abe-0c34-43b8-9ffa-9cb96352cc57" (UID: "5fc02abe-0c34-43b8-9ffa-9cb96352cc57"). InnerVolumeSpecName "kube-api-access-nh7x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:37.542660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:37.538655 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh7x9\" (UniqueName: \"kubernetes.io/projected/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-kube-api-access-nh7x9\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:37.542660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:37.538679 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:37.542660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:37.538715 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc02abe-0c34-43b8-9ffa-9cb96352cc57-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:38.092518 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:38.092490 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn" Oct 30 22:12:38.093291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:38.093010 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn" event={"ID":"5fc02abe-0c34-43b8-9ffa-9cb96352cc57","Type":"ContainerDied","Data":"5d5c9149cfaadccc3836660f4bb8bae50070543a6f7076403548d2ab45f8163e"} Oct 30 22:12:38.093291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:38.093062 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d5c9149cfaadccc3836660f4bb8bae50070543a6f7076403548d2ab45f8163e" Oct 30 22:12:38.094557 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:38.094537 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt" event={"ID":"9b4175f9-5637-46d2-a593-ca01d9b970ed","Type":"ContainerStarted","Data":"bbeb5b28aaa6b292f139c64321f2bcd98355d0218ff38c9f51b17722165d1b71"} Oct 30 22:12:38.242068 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:38.242025 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt" podStartSLOduration=2.242011844 podStartE2EDuration="2.242011844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:38.2270113 +0000 UTC m=+1691.327443032" watchObservedRunningTime="2026-10-30 22:12:38.242011844 +0000 UTC m=+1691.342443576" Oct 30 22:12:38.410676 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:38.403012 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-blzr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 22:12:38.410676 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:38.403088 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" podUID="c4116062-6db3-424f-b4cc-3442ab74a3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 22:12:39.132172 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:39.132132 4335 generic.go:356] "Generic (PLEG): container finished" podID="9b4175f9-5637-46d2-a593-ca01d9b970ed" containerID="bbeb5b28aaa6b292f139c64321f2bcd98355d0218ff38c9f51b17722165d1b71" exitCode=0 Oct 30 22:12:39.132767 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:39.132213 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt" event={"ID":"9b4175f9-5637-46d2-a593-ca01d9b970ed","Type":"ContainerDied","Data":"bbeb5b28aaa6b292f139c64321f2bcd98355d0218ff38c9f51b17722165d1b71"} Oct 30 22:12:39.406035 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:39.405946 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-blzr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 22:12:39.406295 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:39.406265 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" podUID="c4116062-6db3-424f-b4cc-3442ab74a3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 22:12:40.369820 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:40.369792 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt" Oct 30 22:12:40.383922 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:40.383900 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4175f9-5637-46d2-a593-ca01d9b970ed-kube-api-access-fbw4m\" (UniqueName: \"kubernetes.io/projected/9b4175f9-5637-46d2-a593-ca01d9b970ed-kube-api-access-fbw4m\") pod \"9b4175f9-5637-46d2-a593-ca01d9b970ed\" (UID: \"9b4175f9-5637-46d2-a593-ca01d9b970ed\") " Oct 30 22:12:40.384041 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:40.383947 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b4175f9-5637-46d2-a593-ca01d9b970ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4175f9-5637-46d2-a593-ca01d9b970ed-maas-api-xks-ca\") pod \"9b4175f9-5637-46d2-a593-ca01d9b970ed\" (UID: \"9b4175f9-5637-46d2-a593-ca01d9b970ed\") " Oct 30 22:12:40.386140 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:40.386115 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4175f9-5637-46d2-a593-ca01d9b970ed-kube-api-access-fbw4m" pod "9b4175f9-5637-46d2-a593-ca01d9b970ed" (UID: "9b4175f9-5637-46d2-a593-ca01d9b970ed"). InnerVolumeSpecName "kube-api-access-fbw4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:40.386229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:40.386177 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b4175f9-5637-46d2-a593-ca01d9b970ed-maas-api-xks-ca" pod "9b4175f9-5637-46d2-a593-ca01d9b970ed" (UID: "9b4175f9-5637-46d2-a593-ca01d9b970ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:40.423344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:40.413988 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-blzr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 22:12:40.423344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:40.414038 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" podUID="c4116062-6db3-424f-b4cc-3442ab74a3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 22:12:40.485167 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:40.485145 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b4175f9-5637-46d2-a593-ca01d9b970ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4175f9-5637-46d2-a593-ca01d9b970ed-maas-api-openshift-ca\") pod \"9b4175f9-5637-46d2-a593-ca01d9b970ed\" (UID: \"9b4175f9-5637-46d2-a593-ca01d9b970ed\") " Oct 30 22:12:40.485565 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:40.485530 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4175f9-5637-46d2-a593-ca01d9b970ed-maas-api-openshift-ca" pod "9b4175f9-5637-46d2-a593-ca01d9b970ed" (UID: "9b4175f9-5637-46d2-a593-ca01d9b970ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:40.485700 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:40.485609 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbw4m\" (UniqueName: \"kubernetes.io/projected/9b4175f9-5637-46d2-a593-ca01d9b970ed-kube-api-access-fbw4m\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:40.485700 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:40.485649 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4175f9-5637-46d2-a593-ca01d9b970ed-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:40.586779 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:40.586748 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4175f9-5637-46d2-a593-ca01d9b970ed-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:41.156632 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:41.156587 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt" Oct 30 22:12:41.156632 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:41.156599 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt" event={"ID":"9b4175f9-5637-46d2-a593-ca01d9b970ed","Type":"ContainerDied","Data":"1d0fb0e997c016b7af1768d6f4954cfb2fcbfc593a41c6ab975f5c39a0aefd54"} Oct 30 22:12:41.156837 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:41.156674 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d0fb0e997c016b7af1768d6f4954cfb2fcbfc593a41c6ab975f5c39a0aefd54" Oct 30 22:12:41.401383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:41.401297 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-blzr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 22:12:41.401383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:41.401367 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" podUID="c4116062-6db3-424f-b4cc-3442ab74a3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 22:12:42.401772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:42.401732 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-blzr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 22:12:42.402308 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:42.401797 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" podUID="c4116062-6db3-424f-b4cc-3442ab74a3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 22:12:43.405999 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:43.405968 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:43.406397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:43.406369 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:43.406851 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:43.406835 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:12:43.685622 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:43.685541 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq"] Oct 30 22:12:46.689138 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:46.689103 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2"] Oct 30 22:12:46.810997 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:46.810967 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp"] Oct 30 22:12:47.905328 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:47.905295 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97"] Oct 30 22:12:47.906204 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:47.906183 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5fc02abe-0c34-43b8-9ffa-9cb96352cc57" containerName="revoke-keys" Oct 30 22:12:47.906204 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:47.906203 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc02abe-0c34-43b8-9ffa-9cb96352cc57" containerName="revoke-keys" Oct 30 22:12:47.906363 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:47.906246 4335 cpu_manager.go:414] "containerMap: removing container" podUID="9b4175f9-5637-46d2-a593-ca01d9b970ed" containerName="revoke-keys" Oct 30 22:12:47.906363 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:47.906252 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4175f9-5637-46d2-a593-ca01d9b970ed" containerName="revoke-keys" Oct 30 22:12:47.906469 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:47.906373 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc02abe-0c34-43b8-9ffa-9cb96352cc57" containerName="revoke-keys" Oct 30 22:12:47.906469 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:47.906401 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4175f9-5637-46d2-a593-ca01d9b970ed" containerName="revoke-keys" Oct 30 22:12:47.911164 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:47.911116 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97" Oct 30 22:12:47.963488 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:47.963462 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:47.963662 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:47.963460 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:12:47.963662 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:47.963530 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:12:47.967493 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:47.967460 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97"] Oct 30 22:12:48.006167 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:48.006064 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:12:48.031024 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:48.030998 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcrvr\" (UniqueName: \"kubernetes.io/projected/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-kube-api-access-jcrvr\") pod \"maas-api-revoke-sub-a997efcc-n7g97\" (UID: \"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97" Oct 30 22:12:48.031250 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:48.031043 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a997efcc-n7g97\" (UID: \"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97" Oct 30 22:12:48.031367 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:48.031348 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a997efcc-n7g97\" (UID: \"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97" Oct 30 22:12:48.132493 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:48.132467 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a997efcc-n7g97\" (UID: \"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97" Oct 30 22:12:48.132676 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:48.132584 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcrvr\" (UniqueName: \"kubernetes.io/projected/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-kube-api-access-jcrvr\") pod \"maas-api-revoke-sub-a997efcc-n7g97\" (UID: \"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97" Oct 30 22:12:48.132676 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:48.132652 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a997efcc-n7g97\" (UID: \"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97" Oct 30 22:12:48.133131 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:48.133111 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a997efcc-n7g97\" (UID: \"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97" Oct 30 22:12:48.135337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:48.135314 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a997efcc-n7g97\" (UID: \"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97" Oct 30 22:12:48.202678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:48.202583 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcrvr\" (UniqueName: \"kubernetes.io/projected/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-kube-api-access-jcrvr\") pod \"maas-api-revoke-sub-a997efcc-n7g97\" (UID: \"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97" Oct 30 22:12:48.231674 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:48.230047 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97" Oct 30 22:12:48.697638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:48.695104 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq" podUID="368ca2ba-0376-4928-8257-3f9af2de220e" containerName="payload-processing" containerID="cri-o://0b733190948811de462dfcf177a3e677b78696b87c7e1c9d25ee0fe6bb4e5704" gracePeriod=25 Oct 30 22:12:48.833595 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:12:48.832348 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3f076a_bb85_4f0e_bcd9_ef5cc5101516.slice/crio-1e543d3dcd73c452eef3587dbcf9726cbb30e673239b2370785b3f7b8f6cd909 WatchSource:0}: Error finding container 1e543d3dcd73c452eef3587dbcf9726cbb30e673239b2370785b3f7b8f6cd909: Status 404 returned error can't find the container with id 1e543d3dcd73c452eef3587dbcf9726cbb30e673239b2370785b3f7b8f6cd909 Oct 30 22:12:48.836760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:48.836705 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97"] Oct 30 22:12:49.227049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.227029 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq" Oct 30 22:12:49.280146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.280126 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/368ca2ba-0376-4928-8257-3f9af2de220e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/368ca2ba-0376-4928-8257-3f9af2de220e-plugins-config-volume\") pod \"368ca2ba-0376-4928-8257-3f9af2de220e\" (UID: \"368ca2ba-0376-4928-8257-3f9af2de220e\") " Oct 30 22:12:49.280283 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.280270 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/368ca2ba-0376-4928-8257-3f9af2de220e-kube-api-access-z6b9r\" (UniqueName: \"kubernetes.io/projected/368ca2ba-0376-4928-8257-3f9af2de220e-kube-api-access-z6b9r\") pod \"368ca2ba-0376-4928-8257-3f9af2de220e\" (UID: \"368ca2ba-0376-4928-8257-3f9af2de220e\") " Oct 30 22:12:49.280557 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.280524 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/368ca2ba-0376-4928-8257-3f9af2de220e-plugins-config-volume" pod "368ca2ba-0376-4928-8257-3f9af2de220e" (UID: "368ca2ba-0376-4928-8257-3f9af2de220e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:49.280796 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.280777 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/368ca2ba-0376-4928-8257-3f9af2de220e-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:49.282249 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.282230 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/368ca2ba-0376-4928-8257-3f9af2de220e-kube-api-access-z6b9r" pod "368ca2ba-0376-4928-8257-3f9af2de220e" (UID: "368ca2ba-0376-4928-8257-3f9af2de220e"). InnerVolumeSpecName "kube-api-access-z6b9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:49.381802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.381743 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6b9r\" (UniqueName: \"kubernetes.io/projected/368ca2ba-0376-4928-8257-3f9af2de220e-kube-api-access-z6b9r\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:49.422269 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.422237 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97" event={"ID":"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516","Type":"ContainerStarted","Data":"3efe763906edcb384821872f34b635eed10fc99ff2a6311aa4cf4affb106c561"} Oct 30 22:12:49.422440 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.422294 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97" event={"ID":"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516","Type":"ContainerStarted","Data":"1e543d3dcd73c452eef3587dbcf9726cbb30e673239b2370785b3f7b8f6cd909"} Oct 30 22:12:49.424868 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.424842 4335 generic.go:356] "Generic (PLEG): container finished" podID="368ca2ba-0376-4928-8257-3f9af2de220e" containerID="0b733190948811de462dfcf177a3e677b78696b87c7e1c9d25ee0fe6bb4e5704" exitCode=0 Oct 30 22:12:49.424995 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.424897 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq" event={"ID":"368ca2ba-0376-4928-8257-3f9af2de220e","Type":"ContainerDied","Data":"0b733190948811de462dfcf177a3e677b78696b87c7e1c9d25ee0fe6bb4e5704"} Oct 30 22:12:49.424995 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.424938 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq" event={"ID":"368ca2ba-0376-4928-8257-3f9af2de220e","Type":"ContainerDied","Data":"47675aefc07f86c08dd2fb2024657f4b24b81964cba9ec49d71f595d1e3fc67e"} Oct 30 22:12:49.424995 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.424966 4335 scope.go:122] "RemoveContainer" containerID="0b733190948811de462dfcf177a3e677b78696b87c7e1c9d25ee0fe6bb4e5704" Oct 30 22:12:49.425166 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.425127 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq" Oct 30 22:12:49.448456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.448431 4335 scope.go:122] "RemoveContainer" containerID="0b733190948811de462dfcf177a3e677b78696b87c7e1c9d25ee0fe6bb4e5704" Oct 30 22:12:49.448763 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:12:49.448743 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b733190948811de462dfcf177a3e677b78696b87c7e1c9d25ee0fe6bb4e5704\": container with ID starting with 0b733190948811de462dfcf177a3e677b78696b87c7e1c9d25ee0fe6bb4e5704 not found: ID does not exist" containerID="0b733190948811de462dfcf177a3e677b78696b87c7e1c9d25ee0fe6bb4e5704" Oct 30 22:12:49.448843 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.448770 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b733190948811de462dfcf177a3e677b78696b87c7e1c9d25ee0fe6bb4e5704"} err="failed to get container status \"0b733190948811de462dfcf177a3e677b78696b87c7e1c9d25ee0fe6bb4e5704\": rpc error: code = NotFound desc = could not find container \"0b733190948811de462dfcf177a3e677b78696b87c7e1c9d25ee0fe6bb4e5704\": container with ID starting with 0b733190948811de462dfcf177a3e677b78696b87c7e1c9d25ee0fe6bb4e5704 not found: ID does not exist" Oct 30 22:12:49.543852 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.543804 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97" podStartSLOduration=2.543790469 podStartE2EDuration="2.543790469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:49.537374629 +0000 UTC m=+1702.637806362" watchObservedRunningTime="2026-10-30 22:12:49.543790469 +0000 UTC m=+1702.644222226" Oct 30 22:12:49.656472 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.656406 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq"] Oct 30 22:12:49.662742 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:49.662724 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-92ce375b-fb554ffb5-95fqq"] Oct 30 22:12:49.877041 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:12:49.877011 4335 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3f076a_bb85_4f0e_bcd9_ef5cc5101516.slice/crio-3efe763906edcb384821872f34b635eed10fc99ff2a6311aa4cf4affb106c561.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:12:50.439592 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:50.439556 4335 generic.go:356] "Generic (PLEG): container finished" podID="3e3f076a-bb85-4f0e-bcd9-ef5cc5101516" containerID="3efe763906edcb384821872f34b635eed10fc99ff2a6311aa4cf4affb106c561" exitCode=0 Oct 30 22:12:50.440050 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:50.439653 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97" event={"ID":"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516","Type":"ContainerDied","Data":"3efe763906edcb384821872f34b635eed10fc99ff2a6311aa4cf4affb106c561"} Oct 30 22:12:51.060494 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:51.060454 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="368ca2ba-0376-4928-8257-3f9af2de220e" path="/var/lib/kubelet/pods/368ca2ba-0376-4928-8257-3f9af2de220e/volumes" Oct 30 22:12:51.626180 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:51.626153 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97" Oct 30 22:12:51.689717 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:51.689676 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2" podUID="5b3987ee-87d7-445f-a49a-9d62ea088f60" containerName="payload-processing" containerID="cri-o://ba3499ed4177207ad77a436b603315d8678fe15530d1ec67d7a726f56757d365" gracePeriod=25 Oct 30 22:12:51.715099 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:51.715074 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-kube-api-access-jcrvr\" (UniqueName: \"kubernetes.io/projected/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-kube-api-access-jcrvr\") pod \"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516\" (UID: \"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516\") " Oct 30 22:12:51.715197 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:51.715147 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-maas-api-xks-ca\") pod \"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516\" (UID: \"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516\") " Oct 30 22:12:51.715248 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:51.715203 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-maas-api-openshift-ca\") pod \"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516\" (UID: \"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516\") " Oct 30 22:12:51.715566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:51.715533 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-maas-api-openshift-ca" pod "3e3f076a-bb85-4f0e-bcd9-ef5cc5101516" (UID: "3e3f076a-bb85-4f0e-bcd9-ef5cc5101516"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:51.716045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:51.716014 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:51.717870 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:51.717828 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-maas-api-xks-ca" pod "3e3f076a-bb85-4f0e-bcd9-ef5cc5101516" (UID: "3e3f076a-bb85-4f0e-bcd9-ef5cc5101516"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:51.718016 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:51.717992 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-kube-api-access-jcrvr" pod "3e3f076a-bb85-4f0e-bcd9-ef5cc5101516" (UID: "3e3f076a-bb85-4f0e-bcd9-ef5cc5101516"). InnerVolumeSpecName "kube-api-access-jcrvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:51.811965 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:51.811929 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp" podUID="67660558-c6ab-4de8-8f30-80b80c466214" containerName="payload-pre-processing" containerID="cri-o://6d23f27a4c2ff7911a9bcd1bf15afa9b1a7e3fd78e702570c7cb6c5a4486aa64" gracePeriod=25 Oct 30 22:12:51.817423 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:51.817396 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcrvr\" (UniqueName: \"kubernetes.io/projected/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-kube-api-access-jcrvr\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:51.817423 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:51.817425 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:52.284219 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.284189 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2" Oct 30 22:12:52.344640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.344127 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b3987ee-87d7-445f-a49a-9d62ea088f60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3987ee-87d7-445f-a49a-9d62ea088f60-plugins-config-volume\") pod \"5b3987ee-87d7-445f-a49a-9d62ea088f60\" (UID: \"5b3987ee-87d7-445f-a49a-9d62ea088f60\") " Oct 30 22:12:52.344640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.344463 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b3987ee-87d7-445f-a49a-9d62ea088f60-plugins-config-volume" pod "5b3987ee-87d7-445f-a49a-9d62ea088f60" (UID: "5b3987ee-87d7-445f-a49a-9d62ea088f60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:52.344640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.344513 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b3987ee-87d7-445f-a49a-9d62ea088f60-kube-api-access-w27tk\" (UniqueName: \"kubernetes.io/projected/5b3987ee-87d7-445f-a49a-9d62ea088f60-kube-api-access-w27tk\") pod \"5b3987ee-87d7-445f-a49a-9d62ea088f60\" (UID: \"5b3987ee-87d7-445f-a49a-9d62ea088f60\") " Oct 30 22:12:52.345335 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.345310 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3987ee-87d7-445f-a49a-9d62ea088f60-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:52.347061 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.347035 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3987ee-87d7-445f-a49a-9d62ea088f60-kube-api-access-w27tk" pod "5b3987ee-87d7-445f-a49a-9d62ea088f60" (UID: "5b3987ee-87d7-445f-a49a-9d62ea088f60"). InnerVolumeSpecName "kube-api-access-w27tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:52.385367 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.385349 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp" Oct 30 22:12:52.446579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.446550 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67660558-c6ab-4de8-8f30-80b80c466214-kube-api-access-rwr56\" (UniqueName: \"kubernetes.io/projected/67660558-c6ab-4de8-8f30-80b80c466214-kube-api-access-rwr56\") pod \"67660558-c6ab-4de8-8f30-80b80c466214\" (UID: \"67660558-c6ab-4de8-8f30-80b80c466214\") " Oct 30 22:12:52.446735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.446696 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67660558-c6ab-4de8-8f30-80b80c466214-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67660558-c6ab-4de8-8f30-80b80c466214-plugins-config-volume\") pod \"67660558-c6ab-4de8-8f30-80b80c466214\" (UID: \"67660558-c6ab-4de8-8f30-80b80c466214\") " Oct 30 22:12:52.447108 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.447074 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67660558-c6ab-4de8-8f30-80b80c466214-plugins-config-volume" pod "67660558-c6ab-4de8-8f30-80b80c466214" (UID: "67660558-c6ab-4de8-8f30-80b80c466214"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:52.447596 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.447571 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w27tk\" (UniqueName: \"kubernetes.io/projected/5b3987ee-87d7-445f-a49a-9d62ea088f60-kube-api-access-w27tk\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:52.447741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.447601 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67660558-c6ab-4de8-8f30-80b80c466214-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:52.448815 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.448788 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67660558-c6ab-4de8-8f30-80b80c466214-kube-api-access-rwr56" pod "67660558-c6ab-4de8-8f30-80b80c466214" (UID: "67660558-c6ab-4de8-8f30-80b80c466214"). InnerVolumeSpecName "kube-api-access-rwr56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:52.467053 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.467027 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97" Oct 30 22:12:52.467053 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.467031 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97" event={"ID":"3e3f076a-bb85-4f0e-bcd9-ef5cc5101516","Type":"ContainerDied","Data":"1e543d3dcd73c452eef3587dbcf9726cbb30e673239b2370785b3f7b8f6cd909"} Oct 30 22:12:52.467293 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.467083 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e543d3dcd73c452eef3587dbcf9726cbb30e673239b2370785b3f7b8f6cd909" Oct 30 22:12:52.469506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.469297 4335 generic.go:356] "Generic (PLEG): container finished" podID="5b3987ee-87d7-445f-a49a-9d62ea088f60" containerID="ba3499ed4177207ad77a436b603315d8678fe15530d1ec67d7a726f56757d365" exitCode=0 Oct 30 22:12:52.469506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.469345 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2" event={"ID":"5b3987ee-87d7-445f-a49a-9d62ea088f60","Type":"ContainerDied","Data":"ba3499ed4177207ad77a436b603315d8678fe15530d1ec67d7a726f56757d365"} Oct 30 22:12:52.469506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.469401 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2" event={"ID":"5b3987ee-87d7-445f-a49a-9d62ea088f60","Type":"ContainerDied","Data":"44bb4d9526177786c39855d4e3ebdcdef0600279c24c6b6c5c2f705c4760469d"} Oct 30 22:12:52.469506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.469359 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2" Oct 30 22:12:52.469826 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.469433 4335 scope.go:122] "RemoveContainer" containerID="ba3499ed4177207ad77a436b603315d8678fe15530d1ec67d7a726f56757d365" Oct 30 22:12:52.477960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.477921 4335 generic.go:356] "Generic (PLEG): container finished" podID="67660558-c6ab-4de8-8f30-80b80c466214" containerID="6d23f27a4c2ff7911a9bcd1bf15afa9b1a7e3fd78e702570c7cb6c5a4486aa64" exitCode=0 Oct 30 22:12:52.478066 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.477976 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp" Oct 30 22:12:52.478066 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.477987 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp" event={"ID":"67660558-c6ab-4de8-8f30-80b80c466214","Type":"ContainerDied","Data":"6d23f27a4c2ff7911a9bcd1bf15afa9b1a7e3fd78e702570c7cb6c5a4486aa64"} Oct 30 22:12:52.478066 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.478026 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp" event={"ID":"67660558-c6ab-4de8-8f30-80b80c466214","Type":"ContainerDied","Data":"df20ae556449db4c6fdcbc41e7299effff921b5f3d88833f15c45277309dfea6"} Oct 30 22:12:52.493843 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.493824 4335 scope.go:122] "RemoveContainer" containerID="ba3499ed4177207ad77a436b603315d8678fe15530d1ec67d7a726f56757d365" Oct 30 22:12:52.494308 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:12:52.494246 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba3499ed4177207ad77a436b603315d8678fe15530d1ec67d7a726f56757d365\": container with ID starting with ba3499ed4177207ad77a436b603315d8678fe15530d1ec67d7a726f56757d365 not found: ID does not exist" containerID="ba3499ed4177207ad77a436b603315d8678fe15530d1ec67d7a726f56757d365" Oct 30 22:12:52.494308 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.494282 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba3499ed4177207ad77a436b603315d8678fe15530d1ec67d7a726f56757d365"} err="failed to get container status \"ba3499ed4177207ad77a436b603315d8678fe15530d1ec67d7a726f56757d365\": rpc error: code = NotFound desc = could not find container \"ba3499ed4177207ad77a436b603315d8678fe15530d1ec67d7a726f56757d365\": container with ID starting with ba3499ed4177207ad77a436b603315d8678fe15530d1ec67d7a726f56757d365 not found: ID does not exist" Oct 30 22:12:52.494308 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.494303 4335 scope.go:122] "RemoveContainer" containerID="6d23f27a4c2ff7911a9bcd1bf15afa9b1a7e3fd78e702570c7cb6c5a4486aa64" Oct 30 22:12:52.508593 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.508576 4335 scope.go:122] "RemoveContainer" containerID="6d23f27a4c2ff7911a9bcd1bf15afa9b1a7e3fd78e702570c7cb6c5a4486aa64" Oct 30 22:12:52.508946 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:12:52.508923 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d23f27a4c2ff7911a9bcd1bf15afa9b1a7e3fd78e702570c7cb6c5a4486aa64\": container with ID starting with 6d23f27a4c2ff7911a9bcd1bf15afa9b1a7e3fd78e702570c7cb6c5a4486aa64 not found: ID does not exist" containerID="6d23f27a4c2ff7911a9bcd1bf15afa9b1a7e3fd78e702570c7cb6c5a4486aa64" Oct 30 22:12:52.509019 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.508957 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d23f27a4c2ff7911a9bcd1bf15afa9b1a7e3fd78e702570c7cb6c5a4486aa64"} err="failed to get container status \"6d23f27a4c2ff7911a9bcd1bf15afa9b1a7e3fd78e702570c7cb6c5a4486aa64\": rpc error: code = NotFound desc = could not find container \"6d23f27a4c2ff7911a9bcd1bf15afa9b1a7e3fd78e702570c7cb6c5a4486aa64\": container with ID starting with 6d23f27a4c2ff7911a9bcd1bf15afa9b1a7e3fd78e702570c7cb6c5a4486aa64 not found: ID does not exist" Oct 30 22:12:52.549794 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.549766 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwr56\" (UniqueName: \"kubernetes.io/projected/67660558-c6ab-4de8-8f30-80b80c466214-kube-api-access-rwr56\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:52.630796 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.630763 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2"] Oct 30 22:12:52.637288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.637261 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2678f4e1-7f86d64b89-s8tm2"] Oct 30 22:12:52.745760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.745706 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp"] Oct 30 22:12:52.751825 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:52.751802 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2678f4e1-7bcb474df9-76qxp"] Oct 30 22:12:53.062303 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:53.062268 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b3987ee-87d7-445f-a49a-9d62ea088f60" path="/var/lib/kubelet/pods/5b3987ee-87d7-445f-a49a-9d62ea088f60/volumes" Oct 30 22:12:53.063089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:53.063063 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67660558-c6ab-4de8-8f30-80b80c466214" path="/var/lib/kubelet/pods/67660558-c6ab-4de8-8f30-80b80c466214/volumes" Oct 30 22:12:55.330851 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:12:55.330802 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:12:57.537111 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:57.537076 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:12:58.097537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.097504 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p"] Oct 30 22:12:58.098288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.098273 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5b3987ee-87d7-445f-a49a-9d62ea088f60" containerName="payload-processing" Oct 30 22:12:58.098354 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.098290 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b3987ee-87d7-445f-a49a-9d62ea088f60" containerName="payload-processing" Oct 30 22:12:58.098354 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.098308 4335 cpu_manager.go:414] "containerMap: removing container" podUID="67660558-c6ab-4de8-8f30-80b80c466214" containerName="payload-pre-processing" Oct 30 22:12:58.098354 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.098314 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67660558-c6ab-4de8-8f30-80b80c466214" containerName="payload-pre-processing" Oct 30 22:12:58.098354 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.098345 4335 cpu_manager.go:414] "containerMap: removing container" podUID="3e3f076a-bb85-4f0e-bcd9-ef5cc5101516" containerName="revoke-keys" Oct 30 22:12:58.098354 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.098350 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e3f076a-bb85-4f0e-bcd9-ef5cc5101516" containerName="revoke-keys" Oct 30 22:12:58.098537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.098368 4335 cpu_manager.go:414] "containerMap: removing container" podUID="368ca2ba-0376-4928-8257-3f9af2de220e" containerName="payload-processing" Oct 30 22:12:58.098537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.098372 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="368ca2ba-0376-4928-8257-3f9af2de220e" containerName="payload-processing" Oct 30 22:12:58.098537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.098481 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e3f076a-bb85-4f0e-bcd9-ef5cc5101516" containerName="revoke-keys" Oct 30 22:12:58.098537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.098491 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="368ca2ba-0376-4928-8257-3f9af2de220e" containerName="payload-processing" Oct 30 22:12:58.098537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.098497 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b3987ee-87d7-445f-a49a-9d62ea088f60" containerName="payload-processing" Oct 30 22:12:58.098537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.098506 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="67660558-c6ab-4de8-8f30-80b80c466214" containerName="payload-pre-processing" Oct 30 22:12:58.104222 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.104198 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p" Oct 30 22:12:58.227116 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.227074 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p"] Oct 30 22:12:58.234681 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.234654 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:12:58.234836 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.234686 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:58.234836 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.234687 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:12:58.234836 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.234687 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:12:58.241071 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.241035 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de35ce17-45e7-4cca-96dd-0f1a01e80317-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p\" (UID: \"de35ce17-45e7-4cca-96dd-0f1a01e80317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p" Oct 30 22:12:58.241201 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.241084 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de35ce17-45e7-4cca-96dd-0f1a01e80317-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p\" (UID: \"de35ce17-45e7-4cca-96dd-0f1a01e80317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p" Oct 30 22:12:58.241286 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.241260 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv6sl\" (UniqueName: \"kubernetes.io/projected/de35ce17-45e7-4cca-96dd-0f1a01e80317-kube-api-access-sv6sl\") pod \"maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p\" (UID: \"de35ce17-45e7-4cca-96dd-0f1a01e80317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p" Oct 30 22:12:58.343758 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.343698 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de35ce17-45e7-4cca-96dd-0f1a01e80317-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p\" (UID: \"de35ce17-45e7-4cca-96dd-0f1a01e80317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p" Oct 30 22:12:58.343963 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.343761 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de35ce17-45e7-4cca-96dd-0f1a01e80317-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p\" (UID: \"de35ce17-45e7-4cca-96dd-0f1a01e80317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p" Oct 30 22:12:58.343963 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.343853 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv6sl\" (UniqueName: \"kubernetes.io/projected/de35ce17-45e7-4cca-96dd-0f1a01e80317-kube-api-access-sv6sl\") pod \"maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p\" (UID: \"de35ce17-45e7-4cca-96dd-0f1a01e80317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p" Oct 30 22:12:58.344444 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.344419 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de35ce17-45e7-4cca-96dd-0f1a01e80317-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p\" (UID: \"de35ce17-45e7-4cca-96dd-0f1a01e80317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p" Oct 30 22:12:58.346762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.346742 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de35ce17-45e7-4cca-96dd-0f1a01e80317-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p\" (UID: \"de35ce17-45e7-4cca-96dd-0f1a01e80317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p" Oct 30 22:12:58.404749 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.404658 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv6sl\" (UniqueName: \"kubernetes.io/projected/de35ce17-45e7-4cca-96dd-0f1a01e80317-kube-api-access-sv6sl\") pod \"maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p\" (UID: \"de35ce17-45e7-4cca-96dd-0f1a01e80317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p" Oct 30 22:12:58.422073 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.422040 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p" Oct 30 22:12:58.829325 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:58.829291 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p"] Oct 30 22:12:59.560500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:59.560452 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p" event={"ID":"de35ce17-45e7-4cca-96dd-0f1a01e80317","Type":"ContainerStarted","Data":"0f90f7741569279ea0f2f2ef510bb38d5ebb5e26870581ce8cd5f53261cd4dbc"} Oct 30 22:12:59.560707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:59.560516 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p" event={"ID":"de35ce17-45e7-4cca-96dd-0f1a01e80317","Type":"ContainerStarted","Data":"e3a9c066ce50f4a9267ead4a67609cc08cd4b6d9e71d27caec557363dbab2a57"} Oct 30 22:12:59.679072 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:12:59.679024 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p" podStartSLOduration=1.6790089030000002 podStartE2EDuration="1.679008903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:59.675774572 +0000 UTC m=+1712.776206368" watchObservedRunningTime="2026-10-30 22:12:59.679008903 +0000 UTC m=+1712.779440678" Oct 30 22:13:00.572946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:00.572908 4335 generic.go:356] "Generic (PLEG): container finished" podID="de35ce17-45e7-4cca-96dd-0f1a01e80317" containerID="0f90f7741569279ea0f2f2ef510bb38d5ebb5e26870581ce8cd5f53261cd4dbc" exitCode=0 Oct 30 22:13:00.573371 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:00.572993 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p" event={"ID":"de35ce17-45e7-4cca-96dd-0f1a01e80317","Type":"ContainerDied","Data":"0f90f7741569279ea0f2f2ef510bb38d5ebb5e26870581ce8cd5f53261cd4dbc"} Oct 30 22:13:01.831913 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:01.831880 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p" Oct 30 22:13:01.930335 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:01.930259 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de35ce17-45e7-4cca-96dd-0f1a01e80317-kube-api-access-sv6sl\" (UniqueName: \"kubernetes.io/projected/de35ce17-45e7-4cca-96dd-0f1a01e80317-kube-api-access-sv6sl\") pod \"de35ce17-45e7-4cca-96dd-0f1a01e80317\" (UID: \"de35ce17-45e7-4cca-96dd-0f1a01e80317\") " Oct 30 22:13:01.930475 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:01.930338 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de35ce17-45e7-4cca-96dd-0f1a01e80317-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de35ce17-45e7-4cca-96dd-0f1a01e80317-maas-api-openshift-ca\") pod \"de35ce17-45e7-4cca-96dd-0f1a01e80317\" (UID: \"de35ce17-45e7-4cca-96dd-0f1a01e80317\") " Oct 30 22:13:01.930475 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:01.930410 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de35ce17-45e7-4cca-96dd-0f1a01e80317-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de35ce17-45e7-4cca-96dd-0f1a01e80317-maas-api-xks-ca\") pod \"de35ce17-45e7-4cca-96dd-0f1a01e80317\" (UID: \"de35ce17-45e7-4cca-96dd-0f1a01e80317\") " Oct 30 22:13:01.930721 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:01.930692 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de35ce17-45e7-4cca-96dd-0f1a01e80317-maas-api-openshift-ca" pod "de35ce17-45e7-4cca-96dd-0f1a01e80317" (UID: "de35ce17-45e7-4cca-96dd-0f1a01e80317"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:01.931078 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:01.931053 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de35ce17-45e7-4cca-96dd-0f1a01e80317-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:01.932563 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:01.932542 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de35ce17-45e7-4cca-96dd-0f1a01e80317-maas-api-xks-ca" pod "de35ce17-45e7-4cca-96dd-0f1a01e80317" (UID: "de35ce17-45e7-4cca-96dd-0f1a01e80317"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:01.932730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:01.932702 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de35ce17-45e7-4cca-96dd-0f1a01e80317-kube-api-access-sv6sl" pod "de35ce17-45e7-4cca-96dd-0f1a01e80317" (UID: "de35ce17-45e7-4cca-96dd-0f1a01e80317"). InnerVolumeSpecName "kube-api-access-sv6sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:02.033486 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:02.033453 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv6sl\" (UniqueName: \"kubernetes.io/projected/de35ce17-45e7-4cca-96dd-0f1a01e80317-kube-api-access-sv6sl\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:02.033486 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:02.033488 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de35ce17-45e7-4cca-96dd-0f1a01e80317-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:02.593348 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:02.593319 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p" Oct 30 22:13:02.593514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:02.593318 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p" event={"ID":"de35ce17-45e7-4cca-96dd-0f1a01e80317","Type":"ContainerDied","Data":"e3a9c066ce50f4a9267ead4a67609cc08cd4b6d9e71d27caec557363dbab2a57"} Oct 30 22:13:02.593514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:02.593444 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3a9c066ce50f4a9267ead4a67609cc08cd4b6d9e71d27caec557363dbab2a57" Oct 30 22:13:05.060383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:05.060346 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc"] Oct 30 22:13:05.105123 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:05.105098 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019b41c8-cxwjc"] Oct 30 22:13:07.061558 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:07.061525 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc8371e7-7efd-4728-8a3b-82ce85f361c9" path="/var/lib/kubelet/pods/bc8371e7-7efd-4728-8a3b-82ce85f361c9/volumes" Oct 30 22:13:08.684386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:08.684350 4335 generic.go:356] "Generic (PLEG): container finished" podID="52ed8581-628f-42ce-8ad0-8899cce519f2" containerID="931591b0ddb2a6689f4d5772133a0a60027e4ac44619f0bafad46ad29c55ef4b" exitCode=0 Oct 30 22:13:08.684941 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:08.684423 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" event={"ID":"52ed8581-628f-42ce-8ad0-8899cce519f2","Type":"ContainerDied","Data":"931591b0ddb2a6689f4d5772133a0a60027e4ac44619f0bafad46ad29c55ef4b"} Oct 30 22:13:08.684941 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:08.684889 4335 scope.go:122] "RemoveContainer" containerID="931591b0ddb2a6689f4d5772133a0a60027e4ac44619f0bafad46ad29c55ef4b" Oct 30 22:13:08.686684 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:08.686608 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:13:08.686785 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:08.686743 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:13:08.686834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:08.686793 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:13:09.696447 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:09.696410 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" event={"ID":"52ed8581-628f-42ce-8ad0-8899cce519f2","Type":"ContainerStarted","Data":"5c7b0dedde604dbea078fdff940dec42807d6563d5c5ceb3be0949deb9a90722"} Oct 30 22:13:09.829437 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:09.829407 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:13:10.058630 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:10.058587 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s"] Oct 30 22:13:10.069486 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:10.069456 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df78edc-pr26s"] Oct 30 22:13:10.648004 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:10.647973 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:13:10.649413 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:10.649391 4335 patch_prober.go:29] interesting pod/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:13:10.649507 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:10.649445 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" podUID="52ed8581-628f-42ce-8ad0-8899cce519f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 22:13:11.060589 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:11.060560 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da09a9f7-3a8c-42bb-9b45-74ecf64351d8" path="/var/lib/kubelet/pods/da09a9f7-3a8c-42bb-9b45-74ecf64351d8/volumes" Oct 30 22:13:11.651738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:11.651703 4335 patch_prober.go:29] interesting pod/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:13:11.652023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:11.651994 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" podUID="52ed8581-628f-42ce-8ad0-8899cce519f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 22:13:12.647517 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:12.647485 4335 patch_prober.go:29] interesting pod/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:13:12.648001 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:12.647544 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" podUID="52ed8581-628f-42ce-8ad0-8899cce519f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 22:13:13.647347 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:13.647308 4335 patch_prober.go:29] interesting pod/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:13:13.647533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:13.647374 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" podUID="52ed8581-628f-42ce-8ad0-8899cce519f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 22:13:14.647538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:14.647501 4335 patch_prober.go:29] interesting pod/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:13:14.647993 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:14.647570 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" podUID="52ed8581-628f-42ce-8ad0-8899cce519f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 22:13:15.651997 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:15.651967 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:13:15.652975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:15.652956 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:13:19.059178 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:19.059148 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx"] Oct 30 22:13:19.113842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:19.113809 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b56eefa-ndcbx"] Oct 30 22:13:20.057134 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:20.057101 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb"] Oct 30 22:13:20.116490 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:20.116457 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46dd922e-9mnjb"] Oct 30 22:13:21.059772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:21.059742 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b95ec70a-e549-49b4-8160-c2ecabdba2f2" path="/var/lib/kubelet/pods/b95ec70a-e549-49b4-8160-c2ecabdba2f2/volumes" Oct 30 22:13:21.060202 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:21.060187 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb79fe62-f447-40a0-bce0-1472e8ef5b6b" path="/var/lib/kubelet/pods/cb79fe62-f447-40a0-bce0-1472e8ef5b6b/volumes" Oct 30 22:13:21.660017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:21.659943 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt"] Oct 30 22:13:21.694456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:21.694430 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2678f4e1-wfzpt"] Oct 30 22:13:23.059679 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:23.059606 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b4175f9-5637-46d2-a593-ca01d9b970ed" path="/var/lib/kubelet/pods/9b4175f9-5637-46d2-a593-ca01d9b970ed/volumes" Oct 30 22:13:25.322380 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:13:25.322349 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:13:27.983825 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:27.983787 4335 scope.go:122] "RemoveContainer" containerID="2775e31261cb74bef7856a04a29d3d5d2497b0349908a4e79aea3d3f2827d607" Oct 30 22:13:28.102431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:28.102402 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn"] Oct 30 22:13:28.161758 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:28.161715 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ce375b-826xn"] Oct 30 22:13:28.167038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:28.166897 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf"] Oct 30 22:13:28.167390 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:28.167363 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" podUID="27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" containerName="istio-proxy" containerID="cri-o://5491bab0f2a7b3c4bf92a55ecf21b6ba8e5962c8f74931d15c89ff799cc2062e" gracePeriod=30 Oct 30 22:13:29.060197 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:29.060162 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc02abe-0c34-43b8-9ffa-9cb96352cc57" path="/var/lib/kubelet/pods/5fc02abe-0c34-43b8-9ffa-9cb96352cc57/volumes" Oct 30 22:13:30.647129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:30.647099 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq"] Oct 30 22:13:30.647970 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:30.647949 4335 cpu_manager.go:414] "containerMap: removing container" podUID="de35ce17-45e7-4cca-96dd-0f1a01e80317" containerName="revoke-keys" Oct 30 22:13:30.648048 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:30.647972 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de35ce17-45e7-4cca-96dd-0f1a01e80317" containerName="revoke-keys" Oct 30 22:13:30.648113 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:30.648102 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="de35ce17-45e7-4cca-96dd-0f1a01e80317" containerName="revoke-keys" Oct 30 22:13:30.653814 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:30.653795 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq" Oct 30 22:13:30.712883 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:30.712859 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:13:30.713055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:30.713006 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq"] Oct 30 22:13:30.713258 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:30.713240 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:13:30.765633 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:30.761015 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:13:30.765633 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:30.761077 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:13:30.841281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:30.841252 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26dc0b9-623c-4526-849c-4636fba17e35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq\" (UID: \"e26dc0b9-623c-4526-849c-4636fba17e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq" Oct 30 22:13:30.841433 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:30.841412 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxckd\" (UniqueName: \"kubernetes.io/projected/e26dc0b9-623c-4526-849c-4636fba17e35-kube-api-access-xxckd\") pod \"maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq\" (UID: \"e26dc0b9-623c-4526-849c-4636fba17e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq" Oct 30 22:13:30.841694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:30.841673 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26dc0b9-623c-4526-849c-4636fba17e35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq\" (UID: \"e26dc0b9-623c-4526-849c-4636fba17e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq" Oct 30 22:13:30.943550 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:30.943483 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxckd\" (UniqueName: \"kubernetes.io/projected/e26dc0b9-623c-4526-849c-4636fba17e35-kube-api-access-xxckd\") pod \"maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq\" (UID: \"e26dc0b9-623c-4526-849c-4636fba17e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq" Oct 30 22:13:30.943726 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:30.943697 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26dc0b9-623c-4526-849c-4636fba17e35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq\" (UID: \"e26dc0b9-623c-4526-849c-4636fba17e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq" Oct 30 22:13:30.943861 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:30.943843 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26dc0b9-623c-4526-849c-4636fba17e35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq\" (UID: \"e26dc0b9-623c-4526-849c-4636fba17e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq" Oct 30 22:13:30.944492 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:30.944468 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26dc0b9-623c-4526-849c-4636fba17e35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq\" (UID: \"e26dc0b9-623c-4526-849c-4636fba17e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq" Oct 30 22:13:30.946474 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:30.946447 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26dc0b9-623c-4526-849c-4636fba17e35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq\" (UID: \"e26dc0b9-623c-4526-849c-4636fba17e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq" Oct 30 22:13:31.010786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:31.010763 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxckd\" (UniqueName: \"kubernetes.io/projected/e26dc0b9-623c-4526-849c-4636fba17e35-kube-api-access-xxckd\") pod \"maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq\" (UID: \"e26dc0b9-623c-4526-849c-4636fba17e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq" Oct 30 22:13:31.269011 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:31.268940 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq" Oct 30 22:13:31.663585 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:13:31.663555 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26dc0b9_623c_4526_849c_4636fba17e35.slice/crio-4dba288fd990469cc1bd5a88e11fc9851e6134ba775327bfbf07faaf24ababda WatchSource:0}: Error finding container 4dba288fd990469cc1bd5a88e11fc9851e6134ba775327bfbf07faaf24ababda: Status 404 returned error can't find the container with id 4dba288fd990469cc1bd5a88e11fc9851e6134ba775327bfbf07faaf24ababda Oct 30 22:13:31.672407 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:31.672371 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq"] Oct 30 22:13:32.128256 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:32.128229 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq" event={"ID":"e26dc0b9-623c-4526-849c-4636fba17e35","Type":"ContainerStarted","Data":"84603118acc83dd68f48cb41efcee04b7887612a2516670fb0c0eeaae2f39ceb"} Oct 30 22:13:32.128380 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:32.128278 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq" event={"ID":"e26dc0b9-623c-4526-849c-4636fba17e35","Type":"ContainerStarted","Data":"4dba288fd990469cc1bd5a88e11fc9851e6134ba775327bfbf07faaf24ababda"} Oct 30 22:13:32.265937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:32.265863 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq" podStartSLOduration=2.265850838 podStartE2EDuration="2.265850838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:32.2594958 +0000 UTC m=+1745.359927531" watchObservedRunningTime="2026-10-30 22:13:32.265850838 +0000 UTC m=+1745.366282570" Oct 30 22:13:33.138980 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.138948 4335 generic.go:356] "Generic (PLEG): container finished" podID="e26dc0b9-623c-4526-849c-4636fba17e35" containerID="84603118acc83dd68f48cb41efcee04b7887612a2516670fb0c0eeaae2f39ceb" exitCode=0 Oct 30 22:13:33.139399 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.139008 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq" event={"ID":"e26dc0b9-623c-4526-849c-4636fba17e35","Type":"ContainerDied","Data":"84603118acc83dd68f48cb41efcee04b7887612a2516670fb0c0eeaae2f39ceb"} Oct 30 22:13:33.488937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.488911 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:13:33.589467 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.589438 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-kube-api-access-g9zlf\" (UniqueName: \"kubernetes.io/projected/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-kube-api-access-g9zlf\") pod \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " Oct 30 22:13:33.589645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.589549 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-token\" (UniqueName: \"kubernetes.io/projected/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-token\") pod \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " Oct 30 22:13:33.589645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.589582 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-workload-socket\") pod \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " Oct 30 22:13:33.589730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.589647 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-data\") pod \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " Oct 30 22:13:33.589800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.589777 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istiod-ca-cert\") pod \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " Oct 30 22:13:33.589936 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.589913 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-credential-socket\") pod \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " Oct 30 22:13:33.590067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.589914 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-data" pod "27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" (UID: "27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:33.590067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.589929 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-workload-socket" pod "27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" (UID: "27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:33.590067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.590042 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-envoy\") pod \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " Oct 30 22:13:33.590192 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.590084 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-credential-socket" pod "27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" (UID: "27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:33.590192 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.590108 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istiod-ca-cert" pod "27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" (UID: "27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:33.590192 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.590139 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-workload-certs\") pod \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " Oct 30 22:13:33.590323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.590210 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-podinfo\") pod \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\" (UID: \"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0\") " Oct 30 22:13:33.590447 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.590430 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-workload-certs" pod "27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" (UID: "27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:33.591255 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.591231 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.591334 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.591264 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.591334 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.591281 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.591334 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.591294 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.591334 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.591308 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.592327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.592300 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-envoy" pod "27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" (UID: "27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:33.592505 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.592482 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-podinfo" pod "27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" (UID: "27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:13:33.592505 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.592494 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-kube-api-access-g9zlf" pod "27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" (UID: "27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0"). InnerVolumeSpecName "kube-api-access-g9zlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:33.592689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.592525 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-token" pod "27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" (UID: "27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:33.715697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.712110 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.715697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.712145 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9zlf\" (UniqueName: \"kubernetes.io/projected/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-kube-api-access-g9zlf\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.715697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.712163 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.715697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:33.712178 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:34.151394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.151352 4335 generic.go:356] "Generic (PLEG): container finished" podID="27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" containerID="5491bab0f2a7b3c4bf92a55ecf21b6ba8e5962c8f74931d15c89ff799cc2062e" exitCode=0 Oct 30 22:13:34.151394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.151392 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" Oct 30 22:13:34.152035 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.151405 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" event={"ID":"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0","Type":"ContainerDied","Data":"5491bab0f2a7b3c4bf92a55ecf21b6ba8e5962c8f74931d15c89ff799cc2062e"} Oct 30 22:13:34.152035 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.151470 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf" event={"ID":"27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0","Type":"ContainerDied","Data":"961814a7620a326b7f0ecaaf48b3d8524247128c1968503d209061263e504a2c"} Oct 30 22:13:34.152035 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.151509 4335 scope.go:122] "RemoveContainer" containerID="5491bab0f2a7b3c4bf92a55ecf21b6ba8e5962c8f74931d15c89ff799cc2062e" Oct 30 22:13:34.167290 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.167268 4335 scope.go:122] "RemoveContainer" containerID="c379493889432c301d6e169ec821fe5a8d11569eff5d73932e5176d44ad50c51" Oct 30 22:13:34.182860 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.182818 4335 scope.go:122] "RemoveContainer" containerID="5491bab0f2a7b3c4bf92a55ecf21b6ba8e5962c8f74931d15c89ff799cc2062e" Oct 30 22:13:34.183231 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:13:34.183210 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5491bab0f2a7b3c4bf92a55ecf21b6ba8e5962c8f74931d15c89ff799cc2062e\": container with ID starting with 5491bab0f2a7b3c4bf92a55ecf21b6ba8e5962c8f74931d15c89ff799cc2062e not found: ID does not exist" containerID="5491bab0f2a7b3c4bf92a55ecf21b6ba8e5962c8f74931d15c89ff799cc2062e" Oct 30 22:13:34.183328 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.183242 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5491bab0f2a7b3c4bf92a55ecf21b6ba8e5962c8f74931d15c89ff799cc2062e"} err="failed to get container status \"5491bab0f2a7b3c4bf92a55ecf21b6ba8e5962c8f74931d15c89ff799cc2062e\": rpc error: code = NotFound desc = could not find container \"5491bab0f2a7b3c4bf92a55ecf21b6ba8e5962c8f74931d15c89ff799cc2062e\": container with ID starting with 5491bab0f2a7b3c4bf92a55ecf21b6ba8e5962c8f74931d15c89ff799cc2062e not found: ID does not exist" Oct 30 22:13:34.183328 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.183267 4335 scope.go:122] "RemoveContainer" containerID="c379493889432c301d6e169ec821fe5a8d11569eff5d73932e5176d44ad50c51" Oct 30 22:13:34.183586 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:13:34.183562 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c379493889432c301d6e169ec821fe5a8d11569eff5d73932e5176d44ad50c51\": container with ID starting with c379493889432c301d6e169ec821fe5a8d11569eff5d73932e5176d44ad50c51 not found: ID does not exist" containerID="c379493889432c301d6e169ec821fe5a8d11569eff5d73932e5176d44ad50c51" Oct 30 22:13:34.183678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.183594 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c379493889432c301d6e169ec821fe5a8d11569eff5d73932e5176d44ad50c51"} err="failed to get container status \"c379493889432c301d6e169ec821fe5a8d11569eff5d73932e5176d44ad50c51\": rpc error: code = NotFound desc = could not find container \"c379493889432c301d6e169ec821fe5a8d11569eff5d73932e5176d44ad50c51\": container with ID starting with c379493889432c301d6e169ec821fe5a8d11569eff5d73932e5176d44ad50c51 not found: ID does not exist" Oct 30 22:13:34.348426 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.348397 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf"] Oct 30 22:13:34.358293 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.358261 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq" Oct 30 22:13:34.399778 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.399751 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-92ce375b-gw-openshift-default-748d5977fc-d56cf"] Oct 30 22:13:34.424036 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.423984 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e26dc0b9-623c-4526-849c-4636fba17e35-kube-api-access-xxckd\" (UniqueName: \"kubernetes.io/projected/e26dc0b9-623c-4526-849c-4636fba17e35-kube-api-access-xxckd\") pod \"e26dc0b9-623c-4526-849c-4636fba17e35\" (UID: \"e26dc0b9-623c-4526-849c-4636fba17e35\") " Oct 30 22:13:34.424140 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.424037 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e26dc0b9-623c-4526-849c-4636fba17e35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26dc0b9-623c-4526-849c-4636fba17e35-maas-api-xks-ca\") pod \"e26dc0b9-623c-4526-849c-4636fba17e35\" (UID: \"e26dc0b9-623c-4526-849c-4636fba17e35\") " Oct 30 22:13:34.424140 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.424111 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e26dc0b9-623c-4526-849c-4636fba17e35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26dc0b9-623c-4526-849c-4636fba17e35-maas-api-openshift-ca\") pod \"e26dc0b9-623c-4526-849c-4636fba17e35\" (UID: \"e26dc0b9-623c-4526-849c-4636fba17e35\") " Oct 30 22:13:34.424519 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.424482 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e26dc0b9-623c-4526-849c-4636fba17e35-maas-api-openshift-ca" pod "e26dc0b9-623c-4526-849c-4636fba17e35" (UID: "e26dc0b9-623c-4526-849c-4636fba17e35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:34.425343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.425317 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26dc0b9-623c-4526-849c-4636fba17e35-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:34.426416 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.426381 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e26dc0b9-623c-4526-849c-4636fba17e35-maas-api-xks-ca" pod "e26dc0b9-623c-4526-849c-4636fba17e35" (UID: "e26dc0b9-623c-4526-849c-4636fba17e35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:34.426504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.426458 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26dc0b9-623c-4526-849c-4636fba17e35-kube-api-access-xxckd" pod "e26dc0b9-623c-4526-849c-4636fba17e35" (UID: "e26dc0b9-623c-4526-849c-4636fba17e35"). InnerVolumeSpecName "kube-api-access-xxckd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:34.526822 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.526789 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxckd\" (UniqueName: \"kubernetes.io/projected/e26dc0b9-623c-4526-849c-4636fba17e35-kube-api-access-xxckd\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:34.526822 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:34.526819 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26dc0b9-623c-4526-849c-4636fba17e35-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:35.060040 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:35.060008 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" path="/var/lib/kubelet/pods/27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0/volumes" Oct 30 22:13:35.164842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:35.164812 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq" event={"ID":"e26dc0b9-623c-4526-849c-4636fba17e35","Type":"ContainerDied","Data":"4dba288fd990469cc1bd5a88e11fc9851e6134ba775327bfbf07faaf24ababda"} Oct 30 22:13:35.165213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:35.164865 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dba288fd990469cc1bd5a88e11fc9851e6134ba775327bfbf07faaf24ababda" Oct 30 22:13:35.165213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:35.164826 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq" Oct 30 22:13:38.207091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.207009 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9"] Oct 30 22:13:38.208148 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.208127 4335 cpu_manager.go:414] "containerMap: removing container" podUID="e26dc0b9-623c-4526-849c-4636fba17e35" containerName="revoke-keys" Oct 30 22:13:38.208202 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.208150 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e26dc0b9-623c-4526-849c-4636fba17e35" containerName="revoke-keys" Oct 30 22:13:38.208202 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.208197 4335 cpu_manager.go:414] "containerMap: removing container" podUID="27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" containerName="istio-proxy" Oct 30 22:13:38.208269 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.208206 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" containerName="istio-proxy" Oct 30 22:13:38.208269 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.208226 4335 cpu_manager.go:414] "containerMap: removing container" podUID="27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" containerName="istio-proxy" Oct 30 22:13:38.208269 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.208233 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" containerName="istio-proxy" Oct 30 22:13:38.208414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.208403 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" containerName="istio-proxy" Oct 30 22:13:38.208448 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.208418 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="27098ffe-5ffe-4d80-bbe3-d4cd7bda73c0" containerName="istio-proxy" Oct 30 22:13:38.208448 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.208439 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="e26dc0b9-623c-4526-849c-4636fba17e35" containerName="revoke-keys" Oct 30 22:13:38.212857 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.212830 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.311547 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.311505 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9"] Oct 30 22:13:38.331771 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.331739 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rccqw\" (UniqueName: \"kubernetes.io/projected/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-kube-api-access-rccqw\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.333668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.332023 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-data\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.333668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.332382 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-envoy\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.333668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.332708 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-workload-certs\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.333668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.332743 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istiod-ca-cert\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.333668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.332833 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-credential-socket\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.333668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.332867 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-token\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.333668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.332894 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-podinfo\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.333668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.333006 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-workload-socket\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.434906 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.434870 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rccqw\" (UniqueName: \"kubernetes.io/projected/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-kube-api-access-rccqw\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.435069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.434914 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-data\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.435069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.434954 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-envoy\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.435179 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.435079 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-workload-certs\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.435179 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.435106 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istiod-ca-cert\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.435292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.435200 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-credential-socket\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.435292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.435230 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-token\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.435292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.435258 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-podinfo\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.435445 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.435366 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-workload-socket\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.435445 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.435384 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-data\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.435643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.435596 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-credential-socket\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.435719 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.435668 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-workload-socket\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.435848 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.435822 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istiod-ca-cert\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.435976 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.435958 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-workload-certs\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.437925 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.437901 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-envoy\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.438096 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.438076 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-podinfo\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.496574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.496517 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-token\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.496912 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.496890 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rccqw\" (UniqueName: \"kubernetes.io/projected/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-kube-api-access-rccqw\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.527887 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.527852 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:38.977300 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.977255 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:13:38.977439 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.977404 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:13:38.977513 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:38.977496 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:13:39.013040 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:39.012960 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9"] Oct 30 22:13:39.217082 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:39.217044 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" event={"ID":"5fabaab8-b9bc-4346-946c-a7bfd8338dc8","Type":"ContainerStarted","Data":"74f640cfe3f8ceb96096b43b1f2f969790531548293524fed6c8c56ed532bacf"} Oct 30 22:13:39.217758 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:39.217107 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" event={"ID":"5fabaab8-b9bc-4346-946c-a7bfd8338dc8","Type":"ContainerStarted","Data":"1dbfeca05c5d175b1df779a595983f95757772e97411dc840436f0ea95bd4190"} Oct 30 22:13:39.353977 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:39.353922 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" podStartSLOduration=1.353905449 podStartE2EDuration="1.353905449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:39.346887205 +0000 UTC m=+1752.447318937" watchObservedRunningTime="2026-10-30 22:13:39.353905449 +0000 UTC m=+1752.454337181" Oct 30 22:13:39.528544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:39.528514 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:40.532794 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:40.532758 4335 patch_prober.go:29] interesting pod/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 22:13:40.533277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:40.532827 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" podUID="5fabaab8-b9bc-4346-946c-a7bfd8338dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 22:13:41.028559 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:41.028517 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj"] Oct 30 22:13:41.134423 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:41.134389 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m"] Oct 30 22:13:41.571422 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:41.571390 4335 patch_prober.go:29] interesting pod/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 22:13:41.572151 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:41.572121 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" podUID="5fabaab8-b9bc-4346-946c-a7bfd8338dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 22:13:42.053208 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:42.053174 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7"] Oct 30 22:13:42.059101 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:42.059070 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ae275ca-n9xx7"] Oct 30 22:13:42.530696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:42.529787 4335 patch_prober.go:29] interesting pod/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 22:13:42.530696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:42.529846 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" podUID="5fabaab8-b9bc-4346-946c-a7bfd8338dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 22:13:43.079738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:43.079687 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2489bed8-5799-488d-b80b-f2e3c159e3a6" path="/var/lib/kubelet/pods/2489bed8-5799-488d-b80b-f2e3c159e3a6/volumes" Oct 30 22:13:43.531675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:43.528541 4335 patch_prober.go:29] interesting pod/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 22:13:43.531675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:43.528596 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" podUID="5fabaab8-b9bc-4346-946c-a7bfd8338dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 22:13:44.541642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:44.529974 4335 patch_prober.go:29] interesting pod/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 22:13:44.541642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:44.530059 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" podUID="5fabaab8-b9bc-4346-946c-a7bfd8338dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 22:13:45.529043 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:45.529003 4335 patch_prober.go:29] interesting pod/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 22:13:45.529241 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:45.529079 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" podUID="5fabaab8-b9bc-4346-946c-a7bfd8338dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 22:13:45.862639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:45.851531 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p"] Oct 30 22:13:45.891046 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:45.890973 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b8490f04-ltn6p"] Oct 30 22:13:45.915302 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:45.915265 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7"] Oct 30 22:13:45.915811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:45.915775 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" podUID="576a247d-5dc2-43f5-9f67-4b572ea9a072" containerName="istio-proxy" containerID="cri-o://684c28e19754bc3e20270bed0c683050a9f0f2621a501888c2f5e1c126731a77" gracePeriod=30 Oct 30 22:13:46.041686 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.041375 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj" podUID="f996c1c2-bd90-4b41-99b3-08f969d9aba5" containerName="payload-processing" containerID="cri-o://c8b31ae1ada9855d1b1609f5583e7aa7b8765bcb02d5c1d53f9b65cccc227955" gracePeriod=25 Oct 30 22:13:46.141818 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.141663 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m" podUID="4dedd858-dcf5-49e5-83c6-abfbd08e8224" containerName="payload-pre-processing" containerID="cri-o://5261d3c9fde1aecf5a437af6e184786c41d86500f0c3d796eb75ef7c43dde878" gracePeriod=25 Oct 30 22:13:46.493180 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.493135 4335 generic.go:356] "Generic (PLEG): container finished" podID="4dedd858-dcf5-49e5-83c6-abfbd08e8224" containerID="5261d3c9fde1aecf5a437af6e184786c41d86500f0c3d796eb75ef7c43dde878" exitCode=0 Oct 30 22:13:46.493476 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.493450 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m" event={"ID":"4dedd858-dcf5-49e5-83c6-abfbd08e8224","Type":"ContainerDied","Data":"5261d3c9fde1aecf5a437af6e184786c41d86500f0c3d796eb75ef7c43dde878"} Oct 30 22:13:46.507656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.504036 4335 generic.go:356] "Generic (PLEG): container finished" podID="f996c1c2-bd90-4b41-99b3-08f969d9aba5" containerID="c8b31ae1ada9855d1b1609f5583e7aa7b8765bcb02d5c1d53f9b65cccc227955" exitCode=0 Oct 30 22:13:46.507656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.504166 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj" event={"ID":"f996c1c2-bd90-4b41-99b3-08f969d9aba5","Type":"ContainerDied","Data":"c8b31ae1ada9855d1b1609f5583e7aa7b8765bcb02d5c1d53f9b65cccc227955"} Oct 30 22:13:46.527364 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.527342 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj" Oct 30 22:13:46.539526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.539499 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:46.541095 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.541075 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:46.542155 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.542140 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:13:46.583174 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.583146 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m" Oct 30 22:13:46.705078 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.705047 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dedd858-dcf5-49e5-83c6-abfbd08e8224-kube-api-access-4hkd7\" (UniqueName: \"kubernetes.io/projected/4dedd858-dcf5-49e5-83c6-abfbd08e8224-kube-api-access-4hkd7\") pod \"4dedd858-dcf5-49e5-83c6-abfbd08e8224\" (UID: \"4dedd858-dcf5-49e5-83c6-abfbd08e8224\") " Oct 30 22:13:46.705281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.705265 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f996c1c2-bd90-4b41-99b3-08f969d9aba5-kube-api-access-vgrqz\" (UniqueName: \"kubernetes.io/projected/f996c1c2-bd90-4b41-99b3-08f969d9aba5-kube-api-access-vgrqz\") pod \"f996c1c2-bd90-4b41-99b3-08f969d9aba5\" (UID: \"f996c1c2-bd90-4b41-99b3-08f969d9aba5\") " Oct 30 22:13:46.705348 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.705327 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f996c1c2-bd90-4b41-99b3-08f969d9aba5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f996c1c2-bd90-4b41-99b3-08f969d9aba5-plugins-config-volume\") pod \"f996c1c2-bd90-4b41-99b3-08f969d9aba5\" (UID: \"f996c1c2-bd90-4b41-99b3-08f969d9aba5\") " Oct 30 22:13:46.705401 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.705381 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dedd858-dcf5-49e5-83c6-abfbd08e8224-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dedd858-dcf5-49e5-83c6-abfbd08e8224-plugins-config-volume\") pod \"4dedd858-dcf5-49e5-83c6-abfbd08e8224\" (UID: \"4dedd858-dcf5-49e5-83c6-abfbd08e8224\") " Oct 30 22:13:46.706871 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.706839 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f996c1c2-bd90-4b41-99b3-08f969d9aba5-plugins-config-volume" pod "f996c1c2-bd90-4b41-99b3-08f969d9aba5" (UID: "f996c1c2-bd90-4b41-99b3-08f969d9aba5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:46.707024 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.706860 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dedd858-dcf5-49e5-83c6-abfbd08e8224-plugins-config-volume" pod "4dedd858-dcf5-49e5-83c6-abfbd08e8224" (UID: "4dedd858-dcf5-49e5-83c6-abfbd08e8224"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:46.708769 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.708396 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dedd858-dcf5-49e5-83c6-abfbd08e8224-kube-api-access-4hkd7" pod "4dedd858-dcf5-49e5-83c6-abfbd08e8224" (UID: "4dedd858-dcf5-49e5-83c6-abfbd08e8224"). InnerVolumeSpecName "kube-api-access-4hkd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:46.709120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.709081 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f996c1c2-bd90-4b41-99b3-08f969d9aba5-kube-api-access-vgrqz" pod "f996c1c2-bd90-4b41-99b3-08f969d9aba5" (UID: "f996c1c2-bd90-4b41-99b3-08f969d9aba5"). InnerVolumeSpecName "kube-api-access-vgrqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:46.812336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.810258 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgrqz\" (UniqueName: \"kubernetes.io/projected/f996c1c2-bd90-4b41-99b3-08f969d9aba5-kube-api-access-vgrqz\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:46.812336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.810296 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f996c1c2-bd90-4b41-99b3-08f969d9aba5-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:46.812336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.810314 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dedd858-dcf5-49e5-83c6-abfbd08e8224-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:46.812336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:46.810348 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hkd7\" (UniqueName: \"kubernetes.io/projected/4dedd858-dcf5-49e5-83c6-abfbd08e8224-kube-api-access-4hkd7\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:47.063240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:47.063169 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de35ce17-45e7-4cca-96dd-0f1a01e80317" path="/var/lib/kubelet/pods/de35ce17-45e7-4cca-96dd-0f1a01e80317/volumes" Oct 30 22:13:47.515728 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:47.515647 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj" event={"ID":"f996c1c2-bd90-4b41-99b3-08f969d9aba5","Type":"ContainerDied","Data":"173273a5b99ceda1a98fd1583d2a80a339d1813049c930bd3a54f9ff089df7c7"} Oct 30 22:13:47.515902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:47.515657 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj" Oct 30 22:13:47.515902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:47.515720 4335 scope.go:122] "RemoveContainer" containerID="c8b31ae1ada9855d1b1609f5583e7aa7b8765bcb02d5c1d53f9b65cccc227955" Oct 30 22:13:47.517686 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:47.517653 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m" Oct 30 22:13:47.517838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:47.517675 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m" event={"ID":"4dedd858-dcf5-49e5-83c6-abfbd08e8224","Type":"ContainerDied","Data":"d53e3a8bba0072ecfc87632251dd39453c9aee78894aa380c9fc16b11ba22305"} Oct 30 22:13:47.529203 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:47.529186 4335 scope.go:122] "RemoveContainer" containerID="5261d3c9fde1aecf5a437af6e184786c41d86500f0c3d796eb75ef7c43dde878" Oct 30 22:13:47.692868 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:47.692836 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m"] Oct 30 22:13:47.741244 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:47.741211 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa8c82d5-69cbdf9799-l649m"] Oct 30 22:13:47.851734 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:47.851701 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj"] Oct 30 22:13:47.857816 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:47.857791 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aa8c82d5-7c46f486c5-mlrnj"] Oct 30 22:13:49.063609 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:49.063571 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dedd858-dcf5-49e5-83c6-abfbd08e8224" path="/var/lib/kubelet/pods/4dedd858-dcf5-49e5-83c6-abfbd08e8224/volumes" Oct 30 22:13:49.064408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:49.064384 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f996c1c2-bd90-4b41-99b3-08f969d9aba5" path="/var/lib/kubelet/pods/f996c1c2-bd90-4b41-99b3-08f969d9aba5/volumes" Oct 30 22:13:51.404988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.404948 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:13:51.568411 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.568380 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-token\" (UniqueName: \"kubernetes.io/projected/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-token\") pod \"576a247d-5dc2-43f5-9f67-4b572ea9a072\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " Oct 30 22:13:51.568579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.568544 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-credential-socket\") pod \"576a247d-5dc2-43f5-9f67-4b572ea9a072\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " Oct 30 22:13:51.568693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.568581 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-workload-socket\") pod \"576a247d-5dc2-43f5-9f67-4b572ea9a072\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " Oct 30 22:13:51.568693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.568657 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-workload-certs\") pod \"576a247d-5dc2-43f5-9f67-4b572ea9a072\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " Oct 30 22:13:51.568810 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.568764 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/576a247d-5dc2-43f5-9f67-4b572ea9a072-kube-api-access-r29zg\" (UniqueName: \"kubernetes.io/projected/576a247d-5dc2-43f5-9f67-4b572ea9a072-kube-api-access-r29zg\") pod \"576a247d-5dc2-43f5-9f67-4b572ea9a072\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " Oct 30 22:13:51.568861 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.568816 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-data\") pod \"576a247d-5dc2-43f5-9f67-4b572ea9a072\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " Oct 30 22:13:51.569024 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.568926 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-podinfo\") pod \"576a247d-5dc2-43f5-9f67-4b572ea9a072\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " Oct 30 22:13:51.569024 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.568997 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-envoy\") pod \"576a247d-5dc2-43f5-9f67-4b572ea9a072\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " Oct 30 22:13:51.569142 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.569045 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/576a247d-5dc2-43f5-9f67-4b572ea9a072-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576a247d-5dc2-43f5-9f67-4b572ea9a072-istiod-ca-cert\") pod \"576a247d-5dc2-43f5-9f67-4b572ea9a072\" (UID: \"576a247d-5dc2-43f5-9f67-4b572ea9a072\") " Oct 30 22:13:51.569142 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.569111 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-workload-certs" pod "576a247d-5dc2-43f5-9f67-4b572ea9a072" (UID: "576a247d-5dc2-43f5-9f67-4b572ea9a072"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:51.569142 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.569111 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-workload-socket" pod "576a247d-5dc2-43f5-9f67-4b572ea9a072" (UID: "576a247d-5dc2-43f5-9f67-4b572ea9a072"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:51.569317 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.569134 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-credential-socket" pod "576a247d-5dc2-43f5-9f67-4b572ea9a072" (UID: "576a247d-5dc2-43f5-9f67-4b572ea9a072"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:51.570042 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.569563 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-data" pod "576a247d-5dc2-43f5-9f67-4b572ea9a072" (UID: "576a247d-5dc2-43f5-9f67-4b572ea9a072"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:51.570042 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.569736 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/576a247d-5dc2-43f5-9f67-4b572ea9a072-istiod-ca-cert" pod "576a247d-5dc2-43f5-9f67-4b572ea9a072" (UID: "576a247d-5dc2-43f5-9f67-4b572ea9a072"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:51.570042 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.569956 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.570042 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.569978 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.570042 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.569993 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.570042 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.570012 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.573015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.572061 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-podinfo" pod "576a247d-5dc2-43f5-9f67-4b572ea9a072" (UID: "576a247d-5dc2-43f5-9f67-4b572ea9a072"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:13:51.573015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.572152 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-token" pod "576a247d-5dc2-43f5-9f67-4b572ea9a072" (UID: "576a247d-5dc2-43f5-9f67-4b572ea9a072"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:51.573015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.572933 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576a247d-5dc2-43f5-9f67-4b572ea9a072-kube-api-access-r29zg" pod "576a247d-5dc2-43f5-9f67-4b572ea9a072" (UID: "576a247d-5dc2-43f5-9f67-4b572ea9a072"). InnerVolumeSpecName "kube-api-access-r29zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:51.574328 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.574285 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-envoy" pod "576a247d-5dc2-43f5-9f67-4b572ea9a072" (UID: "576a247d-5dc2-43f5-9f67-4b572ea9a072"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:51.578547 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.578523 4335 generic.go:356] "Generic (PLEG): container finished" podID="576a247d-5dc2-43f5-9f67-4b572ea9a072" containerID="684c28e19754bc3e20270bed0c683050a9f0f2621a501888c2f5e1c126731a77" exitCode=0 Oct 30 22:13:51.578754 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.578597 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" Oct 30 22:13:51.578911 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.578607 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" event={"ID":"576a247d-5dc2-43f5-9f67-4b572ea9a072","Type":"ContainerDied","Data":"684c28e19754bc3e20270bed0c683050a9f0f2621a501888c2f5e1c126731a77"} Oct 30 22:13:51.578911 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.578879 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7" event={"ID":"576a247d-5dc2-43f5-9f67-4b572ea9a072","Type":"ContainerDied","Data":"5af978520bb017c127bc4e8a8f299bd2acea3c91c047eaf381603c1898b2bc02"} Oct 30 22:13:51.579093 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.578925 4335 scope.go:122] "RemoveContainer" containerID="684c28e19754bc3e20270bed0c683050a9f0f2621a501888c2f5e1c126731a77" Oct 30 22:13:51.597114 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.597093 4335 scope.go:122] "RemoveContainer" containerID="ca898a0d633cfa4c00ee3552245837019b7e807ce4742781dbc44c08d7df89a6" Oct 30 22:13:51.623590 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.623548 4335 scope.go:122] "RemoveContainer" containerID="684c28e19754bc3e20270bed0c683050a9f0f2621a501888c2f5e1c126731a77" Oct 30 22:13:51.624127 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:13:51.624071 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684c28e19754bc3e20270bed0c683050a9f0f2621a501888c2f5e1c126731a77\": container with ID starting with 684c28e19754bc3e20270bed0c683050a9f0f2621a501888c2f5e1c126731a77 not found: ID does not exist" containerID="684c28e19754bc3e20270bed0c683050a9f0f2621a501888c2f5e1c126731a77" Oct 30 22:13:51.624347 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.624273 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684c28e19754bc3e20270bed0c683050a9f0f2621a501888c2f5e1c126731a77"} err="failed to get container status \"684c28e19754bc3e20270bed0c683050a9f0f2621a501888c2f5e1c126731a77\": rpc error: code = NotFound desc = could not find container \"684c28e19754bc3e20270bed0c683050a9f0f2621a501888c2f5e1c126731a77\": container with ID starting with 684c28e19754bc3e20270bed0c683050a9f0f2621a501888c2f5e1c126731a77 not found: ID does not exist" Oct 30 22:13:51.624347 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.624307 4335 scope.go:122] "RemoveContainer" containerID="ca898a0d633cfa4c00ee3552245837019b7e807ce4742781dbc44c08d7df89a6" Oct 30 22:13:51.624937 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:13:51.624873 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca898a0d633cfa4c00ee3552245837019b7e807ce4742781dbc44c08d7df89a6\": container with ID starting with ca898a0d633cfa4c00ee3552245837019b7e807ce4742781dbc44c08d7df89a6 not found: ID does not exist" containerID="ca898a0d633cfa4c00ee3552245837019b7e807ce4742781dbc44c08d7df89a6" Oct 30 22:13:51.625097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.624904 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca898a0d633cfa4c00ee3552245837019b7e807ce4742781dbc44c08d7df89a6"} err="failed to get container status \"ca898a0d633cfa4c00ee3552245837019b7e807ce4742781dbc44c08d7df89a6\": rpc error: code = NotFound desc = could not find container \"ca898a0d633cfa4c00ee3552245837019b7e807ce4742781dbc44c08d7df89a6\": container with ID starting with ca898a0d633cfa4c00ee3552245837019b7e807ce4742781dbc44c08d7df89a6 not found: ID does not exist" Oct 30 22:13:51.672654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.671778 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.672654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.671803 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r29zg\" (UniqueName: \"kubernetes.io/projected/576a247d-5dc2-43f5-9f67-4b572ea9a072-kube-api-access-r29zg\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.672654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.671815 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.672654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.671823 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576a247d-5dc2-43f5-9f67-4b572ea9a072-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.672654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.671832 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576a247d-5dc2-43f5-9f67-4b572ea9a072-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.757528 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.757496 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7"] Oct 30 22:13:51.819486 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:51.819456 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b8490f04-openshift-default-64b8778957-tkvf7"] Oct 30 22:13:52.067364 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:52.066807 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9"] Oct 30 22:13:52.072329 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:52.072305 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2"] Oct 30 22:13:52.077581 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:52.077559 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59fe371e-jknw9"] Oct 30 22:13:52.088005 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:52.087970 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e55b4d99-b4bx2"] Oct 30 22:13:53.060478 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.060444 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="576a247d-5dc2-43f5-9f67-4b572ea9a072" path="/var/lib/kubelet/pods/576a247d-5dc2-43f5-9f67-4b572ea9a072/volumes" Oct 30 22:13:53.060974 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.060957 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce46877e-24e6-4ac1-8207-cb23b169f3f4" path="/var/lib/kubelet/pods/ce46877e-24e6-4ac1-8207-cb23b169f3f4/volumes" Oct 30 22:13:53.061394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.061379 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd17d98-9598-4da0-a37a-1f260a977070" path="/var/lib/kubelet/pods/dbd17d98-9598-4da0-a37a-1f260a977070/volumes" Oct 30 22:13:53.315354 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.315281 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb"] Oct 30 22:13:53.316011 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.315996 4335 cpu_manager.go:414] "containerMap: removing container" podUID="4dedd858-dcf5-49e5-83c6-abfbd08e8224" containerName="payload-pre-processing" Oct 30 22:13:53.316063 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.316013 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dedd858-dcf5-49e5-83c6-abfbd08e8224" containerName="payload-pre-processing" Oct 30 22:13:53.316063 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.316033 4335 cpu_manager.go:414] "containerMap: removing container" podUID="576a247d-5dc2-43f5-9f67-4b572ea9a072" containerName="istio-proxy" Oct 30 22:13:53.316063 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.316038 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="576a247d-5dc2-43f5-9f67-4b572ea9a072" containerName="istio-proxy" Oct 30 22:13:53.316063 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.316045 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f996c1c2-bd90-4b41-99b3-08f969d9aba5" containerName="payload-processing" Oct 30 22:13:53.316063 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.316050 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f996c1c2-bd90-4b41-99b3-08f969d9aba5" containerName="payload-processing" Oct 30 22:13:53.316212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.316072 4335 cpu_manager.go:414] "containerMap: removing container" podUID="576a247d-5dc2-43f5-9f67-4b572ea9a072" containerName="istio-proxy" Oct 30 22:13:53.316212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.316077 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="576a247d-5dc2-43f5-9f67-4b572ea9a072" containerName="istio-proxy" Oct 30 22:13:53.316212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.316182 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="576a247d-5dc2-43f5-9f67-4b572ea9a072" containerName="istio-proxy" Oct 30 22:13:53.316212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.316188 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f996c1c2-bd90-4b41-99b3-08f969d9aba5" containerName="payload-processing" Oct 30 22:13:53.316212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.316200 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="576a247d-5dc2-43f5-9f67-4b572ea9a072" containerName="istio-proxy" Oct 30 22:13:53.316212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.316212 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dedd858-dcf5-49e5-83c6-abfbd08e8224" containerName="payload-pre-processing" Oct 30 22:13:53.321882 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.321863 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.396350 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.396321 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-envoy\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.396481 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.396357 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-workload-socket\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.396481 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.396381 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-token\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.396567 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.396521 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-data\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.396725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.396703 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istiod-ca-cert\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.396796 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.396755 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-workload-certs\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.396861 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.396839 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z85qj\" (UniqueName: \"kubernetes.io/projected/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-kube-api-access-z85qj\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.396911 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.396891 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-podinfo\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.396977 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.396956 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-credential-socket\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.422962 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.422938 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb"] Oct 30 22:13:53.498776 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.498741 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-data\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.498947 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.498867 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istiod-ca-cert\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.498947 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.498914 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-workload-certs\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.499066 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.498988 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z85qj\" (UniqueName: \"kubernetes.io/projected/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-kube-api-access-z85qj\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.499066 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.499045 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-podinfo\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.499177 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.499076 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-credential-socket\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.499235 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.499194 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-data\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.499235 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.499201 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-envoy\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.499344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.499323 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-workload-certs\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.499557 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.499259 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-workload-socket\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.499683 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.499549 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-credential-socket\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.499683 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.499609 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-token\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.499802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.499723 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-workload-socket\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.499899 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.499866 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istiod-ca-cert\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.502301 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.502277 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-envoy\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.502658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.502595 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-podinfo\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.559957 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.559927 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-token\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.560048 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.560002 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z85qj\" (UniqueName: \"kubernetes.io/projected/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-kube-api-access-z85qj\") pod \"e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:53.639302 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:53.639223 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:54.085116 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:13:54.085093 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6bcc09d_5fcd_45d6_86e6_83bd39d98370.slice/crio-272c0def75a2f5ec3f7429e84b76c29fa1f9caeb48aae51f6a7aa59d87b08107 WatchSource:0}: Error finding container 272c0def75a2f5ec3f7429e84b76c29fa1f9caeb48aae51f6a7aa59d87b08107: Status 404 returned error can't find the container with id 272c0def75a2f5ec3f7429e84b76c29fa1f9caeb48aae51f6a7aa59d87b08107 Oct 30 22:13:54.087571 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:54.087547 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb"] Oct 30 22:13:54.088133 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:54.088106 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:13:54.088174 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:54.088166 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:13:54.088205 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:54.088197 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:13:54.613630 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:54.613583 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" event={"ID":"b6bcc09d-5fcd-45d6-86e6-83bd39d98370","Type":"ContainerStarted","Data":"d656f97c9f315710c3013dda4fd5c5a9b59f3478578c3757eb9502ebda00a912"} Oct 30 22:13:54.613798 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:54.613667 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" event={"ID":"b6bcc09d-5fcd-45d6-86e6-83bd39d98370","Type":"ContainerStarted","Data":"272c0def75a2f5ec3f7429e84b76c29fa1f9caeb48aae51f6a7aa59d87b08107"} Oct 30 22:13:54.640119 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:54.640095 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:13:54.838998 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:54.838939 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" podStartSLOduration=1.838919254 podStartE2EDuration="1.838919254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:54.782875941 +0000 UTC m=+1767.883307674" watchObservedRunningTime="2026-10-30 22:13:54.838919254 +0000 UTC m=+1767.939350987" Oct 30 22:13:55.355956 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:13:55.355916 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:13:55.641748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:55.641669 4335 patch_prober.go:29] interesting pod/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 22:13:55.641748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:55.641724 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" podUID="b6bcc09d-5fcd-45d6-86e6-83bd39d98370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 22:13:56.640090 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:56.640052 4335 patch_prober.go:29] interesting pod/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 22:13:56.640485 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:56.640121 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" podUID="b6bcc09d-5fcd-45d6-86e6-83bd39d98370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 22:13:57.639547 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:57.639499 4335 patch_prober.go:29] interesting pod/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 22:13:57.639764 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:57.639564 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" podUID="b6bcc09d-5fcd-45d6-86e6-83bd39d98370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 22:13:58.133660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:58.132427 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8"] Oct 30 22:13:58.140820 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:58.140784 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273e4030-8mwd8"] Oct 30 22:13:58.639943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:58.639909 4335 patch_prober.go:29] interesting pod/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 22:13:58.640153 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:58.639978 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" podUID="b6bcc09d-5fcd-45d6-86e6-83bd39d98370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 22:13:59.060689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:59.060653 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e699390-82de-4cc7-9eb2-14f627713ca0" path="/var/lib/kubelet/pods/4e699390-82de-4cc7-9eb2-14f627713ca0/volumes" Oct 30 22:13:59.640579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:59.640542 4335 patch_prober.go:29] interesting pod/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 22:13:59.640992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:13:59.640598 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" podUID="b6bcc09d-5fcd-45d6-86e6-83bd39d98370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 22:14:00.644580 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:00.644548 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:14:00.645141 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:00.645122 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:14:00.645773 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:00.645755 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:14:03.060833 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:03.060803 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx"] Oct 30 22:14:03.107968 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:03.107935 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-700438b0-7jdwx"] Oct 30 22:14:05.061223 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:05.061187 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb2f107-516a-4fbc-8d31-512df067bba1" path="/var/lib/kubelet/pods/3fb2f107-516a-4fbc-8d31-512df067bba1/volumes" Oct 30 22:14:08.112531 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:08.112494 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn"] Oct 30 22:14:08.118984 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:08.118956 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929a9e57-jkgxn"] Oct 30 22:14:09.060562 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:09.060519 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75828d3a-9c23-4319-bed3-eee5091cbf6d" path="/var/lib/kubelet/pods/75828d3a-9c23-4319-bed3-eee5091cbf6d/volumes" Oct 30 22:14:12.289136 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:12.289098 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq"] Oct 30 22:14:12.329752 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:12.329716 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa8c82d5-x26gq"] Oct 30 22:14:13.059973 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:13.059941 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e26dc0b9-623c-4526-849c-4636fba17e35" path="/var/lib/kubelet/pods/e26dc0b9-623c-4526-849c-4636fba17e35/volumes" Oct 30 22:14:13.060313 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:13.060298 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj"] Oct 30 22:14:13.104692 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:13.104654 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d8c9d7-r4hcj"] Oct 30 22:14:15.061186 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:15.061148 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b5c2b3-a188-48ea-9352-9cb449f19335" path="/var/lib/kubelet/pods/d6b5c2b3-a188-48ea-9352-9cb449f19335/volumes" Oct 30 22:14:15.061591 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:15.061580 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws"] Oct 30 22:14:15.104722 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:15.104695 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d944301f-bbrws"] Oct 30 22:14:17.068405 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:17.068367 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8797e804-75ad-40d8-87c8-dcb6152e9af8" path="/var/lib/kubelet/pods/8797e804-75ad-40d8-87c8-dcb6152e9af8/volumes" Oct 30 22:14:18.051264 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:18.051232 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc"] Oct 30 22:14:18.101221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:18.101178 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86116beb-9vvzc"] Oct 30 22:14:19.067828 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:19.067796 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="952bbcca-025c-40de-be76-21b90316c51a" path="/var/lib/kubelet/pods/952bbcca-025c-40de-be76-21b90316c51a/volumes" Oct 30 22:14:19.888737 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:19.888705 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j"] Oct 30 22:14:19.893573 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:19.893547 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.032045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.032015 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476ce668-42e8-41b9-aea8-9727298c9597-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.032295 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.032275 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.032393 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.032339 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476ce668-42e8-41b9-aea8-9727298c9597-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.032455 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.032389 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476ce668-42e8-41b9-aea8-9727298c9597-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.032510 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.032462 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.032564 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.032521 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.032687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.032628 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.032687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.032658 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.032822 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.032806 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28nbr\" (UniqueName: \"kubernetes.io/projected/476ce668-42e8-41b9-aea8-9727298c9597-kube-api-access-28nbr\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.097784 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.097749 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j"] Oct 30 22:14:20.134635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.134588 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.134800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.134679 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476ce668-42e8-41b9-aea8-9727298c9597-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.134800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.134734 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476ce668-42e8-41b9-aea8-9727298c9597-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.134934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.134803 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.134934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.134858 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.135054 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.134936 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.135054 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.134965 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.135165 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.135097 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28nbr\" (UniqueName: \"kubernetes.io/projected/476ce668-42e8-41b9-aea8-9727298c9597-kube-api-access-28nbr\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.135224 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.135180 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476ce668-42e8-41b9-aea8-9727298c9597-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.135969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.135757 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.136238 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.136211 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.136586 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.136558 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476ce668-42e8-41b9-aea8-9727298c9597-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.137004 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.136962 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.137434 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.137416 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.139845 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.139773 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476ce668-42e8-41b9-aea8-9727298c9597-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.140540 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.140452 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.217146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.216869 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476ce668-42e8-41b9-aea8-9727298c9597-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.239429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.238066 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28nbr\" (UniqueName: \"kubernetes.io/projected/476ce668-42e8-41b9-aea8-9727298c9597-kube-api-access-28nbr\") pod \"maas-default-gateway-openshift-default-6cc458d887-qt88j\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.513767 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.513681 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:20.960856 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:14:20.960827 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476ce668_42e8_41b9_aea8_9727298c9597.slice/crio-c044ba54cb0f51d36f4e66778def9f627b31540ebe2538bf3167fe5c6dcdd9e0 WatchSource:0}: Error finding container c044ba54cb0f51d36f4e66778def9f627b31540ebe2538bf3167fe5c6dcdd9e0: Status 404 returned error can't find the container with id c044ba54cb0f51d36f4e66778def9f627b31540ebe2538bf3167fe5c6dcdd9e0 Oct 30 22:14:20.961651 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.961600 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j"] Oct 30 22:14:20.963713 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.963682 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:14:20.963804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.963745 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:14:20.963804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:20.963774 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:14:21.959938 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:21.959899 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" event={"ID":"476ce668-42e8-41b9-aea8-9727298c9597","Type":"ContainerStarted","Data":"6d755986d7ec37b7cbc8d0576d130ec37f50df3f7b1b146c8ae61f9ee6e0d91d"} Oct 30 22:14:21.960106 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:21.960072 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" event={"ID":"476ce668-42e8-41b9-aea8-9727298c9597","Type":"ContainerStarted","Data":"c044ba54cb0f51d36f4e66778def9f627b31540ebe2538bf3167fe5c6dcdd9e0"} Oct 30 22:14:22.192373 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:22.192322 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" podStartSLOduration=3.192307796 podStartE2EDuration="3.192307796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:22.139311578 +0000 UTC m=+1795.239743312" watchObservedRunningTime="2026-10-30 22:14:22.192307796 +0000 UTC m=+1795.292739528" Oct 30 22:14:22.518639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:22.517032 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:22.526700 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:22.526077 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-qt88j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:14:22.526700 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:22.526149 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" podUID="476ce668-42e8-41b9-aea8-9727298c9597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 22:14:22.871780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:22.871731 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx"] Oct 30 22:14:22.880972 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:22.872049 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" podUID="52ed8581-628f-42ce-8ad0-8899cce519f2" containerName="istio-proxy" containerID="cri-o://5c7b0dedde604dbea078fdff940dec42807d6563d5c5ceb3be0949deb9a90722" gracePeriod=30 Oct 30 22:14:22.893658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:22.893597 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb"] Oct 30 22:14:22.893922 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:22.893873 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" podUID="b6bcc09d-5fcd-45d6-86e6-83bd39d98370" containerName="istio-proxy" containerID="cri-o://d656f97c9f315710c3013dda4fd5c5a9b59f3478578c3757eb9502ebda00a912" gracePeriod=30 Oct 30 22:14:22.966310 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:22.966270 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9"] Oct 30 22:14:22.966658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:22.966585 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" podUID="5fabaab8-b9bc-4346-946c-a7bfd8338dc8" containerName="istio-proxy" containerID="cri-o://74f640cfe3f8ceb96096b43b1f2f969790531548293524fed6c8c56ed532bacf" gracePeriod=30 Oct 30 22:14:23.529637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:23.529190 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-qt88j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:14:23.529637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:23.529250 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" podUID="476ce668-42e8-41b9-aea8-9727298c9597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 22:14:24.053289 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:24.053254 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2"] Oct 30 22:14:24.059122 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:24.059098 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed3da06-wh9q2"] Oct 30 22:14:24.591638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:24.581399 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-qt88j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:14:24.591638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:24.581469 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" podUID="476ce668-42e8-41b9-aea8-9727298c9597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 22:14:25.074871 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:25.074823 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb" path="/var/lib/kubelet/pods/1f6b1c09-4a5d-403c-a6e9-e4b52bf503bb/volumes" Oct 30 22:14:25.365898 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:14:25.365825 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:14:25.522090 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:25.522050 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-qt88j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:14:25.523237 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:25.522812 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" podUID="476ce668-42e8-41b9-aea8-9727298c9597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 22:14:25.750691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:25.748398 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz"] Oct 30 22:14:25.765926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:25.765896 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz" Oct 30 22:14:25.790636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:25.780175 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94191978-6460-43f3-9799-42c7c6286b2a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz\" (UID: \"94191978-6460-43f3-9799-42c7c6286b2a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz" Oct 30 22:14:25.790636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:25.780226 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqhvf\" (UniqueName: \"kubernetes.io/projected/94191978-6460-43f3-9799-42c7c6286b2a-kube-api-access-gqhvf\") pod \"payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz\" (UID: \"94191978-6460-43f3-9799-42c7c6286b2a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz" Oct 30 22:14:25.980429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:25.980392 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2d8852\"" Oct 30 22:14:25.980647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:25.980558 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2d8852-dockercfg-tkcp8\"" Oct 30 22:14:25.987647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:25.981590 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz"] Oct 30 22:14:25.994828 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:25.994780 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94191978-6460-43f3-9799-42c7c6286b2a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz\" (UID: \"94191978-6460-43f3-9799-42c7c6286b2a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz" Oct 30 22:14:25.995086 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:25.995051 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqhvf\" (UniqueName: \"kubernetes.io/projected/94191978-6460-43f3-9799-42c7c6286b2a-kube-api-access-gqhvf\") pod \"payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz\" (UID: \"94191978-6460-43f3-9799-42c7c6286b2a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz" Oct 30 22:14:25.996683 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:25.996606 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94191978-6460-43f3-9799-42c7c6286b2a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz\" (UID: \"94191978-6460-43f3-9799-42c7c6286b2a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz" Oct 30 22:14:26.155817 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:26.142408 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqhvf\" (UniqueName: \"kubernetes.io/projected/94191978-6460-43f3-9799-42c7c6286b2a-kube-api-access-gqhvf\") pod \"payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz\" (UID: \"94191978-6460-43f3-9799-42c7c6286b2a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz" Oct 30 22:14:26.172337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:26.172310 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm"] Oct 30 22:14:26.180852 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:26.180822 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b632cbaa-l9ndm"] Oct 30 22:14:26.214559 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:26.214526 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz" Oct 30 22:14:26.524496 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:26.523965 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-qt88j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:14:26.524496 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:26.524027 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" podUID="476ce668-42e8-41b9-aea8-9727298c9597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 22:14:26.685218 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:14:26.685161 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94191978_6460_43f3_9799_42c7c6286b2a.slice/crio-e1ee23b5e62f5047ed6344e18d8515cef8e24b7ff11a2f405d2dbb5834c40c0c WatchSource:0}: Error finding container e1ee23b5e62f5047ed6344e18d8515cef8e24b7ff11a2f405d2dbb5834c40c0c: Status 404 returned error can't find the container with id e1ee23b5e62f5047ed6344e18d8515cef8e24b7ff11a2f405d2dbb5834c40c0c Oct 30 22:14:26.686923 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:26.686821 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz"] Oct 30 22:14:27.062198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:27.062167 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aada93e-1b88-4bae-993e-fdc80c3227b9" path="/var/lib/kubelet/pods/5aada93e-1b88-4bae-993e-fdc80c3227b9/volumes" Oct 30 22:14:27.189108 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:27.189076 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz" event={"ID":"94191978-6460-43f3-9799-42c7c6286b2a","Type":"ContainerStarted","Data":"f174e2cd9d2ffbc97d9bcd81d8345ede1bc7a9ec25ede5c39d0a2abffdbcf158"} Oct 30 22:14:27.189268 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:27.189148 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz" event={"ID":"94191978-6460-43f3-9799-42c7c6286b2a","Type":"ContainerStarted","Data":"e1ee23b5e62f5047ed6344e18d8515cef8e24b7ff11a2f405d2dbb5834c40c0c"} Oct 30 22:14:27.189268 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:27.189168 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz" Oct 30 22:14:27.314198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:27.314095 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz" podStartSLOduration=2.314076721 podStartE2EDuration="2.314076721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:27.306551548 +0000 UTC m=+1800.406983281" watchObservedRunningTime="2026-10-30 22:14:27.314076721 +0000 UTC m=+1800.414508455" Oct 30 22:14:27.514416 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:27.514371 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-qt88j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:14:27.514589 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:27.514454 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" podUID="476ce668-42e8-41b9-aea8-9727298c9597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 22:14:28.065028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.064896 4335 scope.go:122] "RemoveContainer" containerID="50a0a6363276f011b13bec0ce6d5fdfdae89c078dafc77bf57b86f94528bd9da" Oct 30 22:14:28.086583 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.086548 4335 scope.go:122] "RemoveContainer" containerID="2c0bf7e6e920486774799aca40d09dce8fe793b1b7441a7c7a36f357d3bc5488" Oct 30 22:14:28.123006 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.119539 4335 scope.go:122] "RemoveContainer" containerID="4936d019327908a587624bfb942ab6e8be0259e74feb0c8d5fef68ed8187667f" Oct 30 22:14:28.162059 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.161998 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:14:28.166164 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:14:28.166132 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c0bf7e6e920486774799aca40d09dce8fe793b1b7441a7c7a36f357d3bc5488\": container with ID starting with 2c0bf7e6e920486774799aca40d09dce8fe793b1b7441a7c7a36f357d3bc5488 not found: ID does not exist" containerID="2c0bf7e6e920486774799aca40d09dce8fe793b1b7441a7c7a36f357d3bc5488" Oct 30 22:14:28.166348 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.166327 4335 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"2c0bf7e6e920486774799aca40d09dce8fe793b1b7441a7c7a36f357d3bc5488\": container with ID starting with 2c0bf7e6e920486774799aca40d09dce8fe793b1b7441a7c7a36f357d3bc5488 not found: ID does not exist" Oct 30 22:14:28.240238 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.239901 4335 scope.go:122] "RemoveContainer" containerID="3be0695f35c750aa6959979e3fffa59de18d254cd6c5877017f3b01e8e4e5a5d" Oct 30 22:14:28.247645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.247597 4335 generic.go:356] "Generic (PLEG): container finished" podID="52ed8581-628f-42ce-8ad0-8899cce519f2" containerID="5c7b0dedde604dbea078fdff940dec42807d6563d5c5ceb3be0949deb9a90722" exitCode=0 Oct 30 22:14:28.247943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.247830 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" event={"ID":"52ed8581-628f-42ce-8ad0-8899cce519f2","Type":"ContainerDied","Data":"5c7b0dedde604dbea078fdff940dec42807d6563d5c5ceb3be0949deb9a90722"} Oct 30 22:14:28.267863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.249749 4335 scope.go:122] "RemoveContainer" containerID="931591b0ddb2a6689f4d5772133a0a60027e4ac44619f0bafad46ad29c55ef4b" Oct 30 22:14:28.287297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.287114 4335 generic.go:356] "Generic (PLEG): container finished" podID="b6bcc09d-5fcd-45d6-86e6-83bd39d98370" containerID="d656f97c9f315710c3013dda4fd5c5a9b59f3478578c3757eb9502ebda00a912" exitCode=0 Oct 30 22:14:28.287297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.287222 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" event={"ID":"b6bcc09d-5fcd-45d6-86e6-83bd39d98370","Type":"ContainerDied","Data":"d656f97c9f315710c3013dda4fd5c5a9b59f3478578c3757eb9502ebda00a912"} Oct 30 22:14:28.291702 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.291433 4335 generic.go:356] "Generic (PLEG): container finished" podID="5fabaab8-b9bc-4346-946c-a7bfd8338dc8" containerID="74f640cfe3f8ceb96096b43b1f2f969790531548293524fed6c8c56ed532bacf" exitCode=0 Oct 30 22:14:28.291905 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.291841 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" event={"ID":"5fabaab8-b9bc-4346-946c-a7bfd8338dc8","Type":"ContainerDied","Data":"74f640cfe3f8ceb96096b43b1f2f969790531548293524fed6c8c56ed532bacf"} Oct 30 22:14:28.322982 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.322958 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:14:28.425678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.424465 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-kube-api-access-z85qj\" (UniqueName: \"kubernetes.io/projected/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-kube-api-access-z85qj\") pod \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " Oct 30 22:14:28.425678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.424583 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istiod-ca-cert\") pod \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " Oct 30 22:14:28.425678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.424644 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-workload-certs\") pod \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " Oct 30 22:14:28.425678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.424690 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-envoy\") pod \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " Oct 30 22:14:28.425678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.424718 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-podinfo\") pod \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " Oct 30 22:14:28.425678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.424769 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-workload-socket\") pod \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " Oct 30 22:14:28.425678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.424842 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-credential-socket\") pod \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " Oct 30 22:14:28.425678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.424883 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-token\" (UniqueName: \"kubernetes.io/projected/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-token\") pod \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " Oct 30 22:14:28.425678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.425114 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-data\") pod \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\" (UID: \"b6bcc09d-5fcd-45d6-86e6-83bd39d98370\") " Oct 30 22:14:28.425678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.425550 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-workload-socket" pod "b6bcc09d-5fcd-45d6-86e6-83bd39d98370" (UID: "b6bcc09d-5fcd-45d6-86e6-83bd39d98370"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:28.425678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.425553 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istiod-ca-cert" pod "b6bcc09d-5fcd-45d6-86e6-83bd39d98370" (UID: "b6bcc09d-5fcd-45d6-86e6-83bd39d98370"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:28.426406 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.425748 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-workload-certs" pod "b6bcc09d-5fcd-45d6-86e6-83bd39d98370" (UID: "b6bcc09d-5fcd-45d6-86e6-83bd39d98370"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:28.426406 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.426184 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-credential-socket" pod "b6bcc09d-5fcd-45d6-86e6-83bd39d98370" (UID: "b6bcc09d-5fcd-45d6-86e6-83bd39d98370"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:28.426537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.426429 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.426537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.426447 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.426537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.426461 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.426537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.426475 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.427448 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.427415 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-data" pod "b6bcc09d-5fcd-45d6-86e6-83bd39d98370" (UID: "b6bcc09d-5fcd-45d6-86e6-83bd39d98370"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:28.436758 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.434952 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-podinfo" pod "b6bcc09d-5fcd-45d6-86e6-83bd39d98370" (UID: "b6bcc09d-5fcd-45d6-86e6-83bd39d98370"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:28.436758 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.435974 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-envoy" pod "b6bcc09d-5fcd-45d6-86e6-83bd39d98370" (UID: "b6bcc09d-5fcd-45d6-86e6-83bd39d98370"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:28.436758 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.436090 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-kube-api-access-z85qj" pod "b6bcc09d-5fcd-45d6-86e6-83bd39d98370" (UID: "b6bcc09d-5fcd-45d6-86e6-83bd39d98370"). InnerVolumeSpecName "kube-api-access-z85qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:28.441181 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.437892 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-token" pod "b6bcc09d-5fcd-45d6-86e6-83bd39d98370" (UID: "b6bcc09d-5fcd-45d6-86e6-83bd39d98370"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:28.450004 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.449984 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:14:28.450663 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.450642 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:14:28.514524 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.514487 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-qt88j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:14:28.515143 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.514786 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" podUID="476ce668-42e8-41b9-aea8-9727298c9597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 22:14:28.528875 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.528844 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-token\" (UniqueName: \"kubernetes.io/projected/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-token\") pod \"52ed8581-628f-42ce-8ad0-8899cce519f2\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " Oct 30 22:14:28.529019 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.528906 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-podinfo\") pod \"52ed8581-628f-42ce-8ad0-8899cce519f2\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " Oct 30 22:14:28.529019 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.529012 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-envoy\") pod \"52ed8581-628f-42ce-8ad0-8899cce519f2\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " Oct 30 22:14:28.529147 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.529092 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-credential-socket\") pod \"52ed8581-628f-42ce-8ad0-8899cce519f2\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " Oct 30 22:14:28.529210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.529198 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-data\") pod \"52ed8581-628f-42ce-8ad0-8899cce519f2\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " Oct 30 22:14:28.529277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.529247 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-workload-socket\") pod \"52ed8581-628f-42ce-8ad0-8899cce519f2\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " Oct 30 22:14:28.529343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.529305 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ed8581-628f-42ce-8ad0-8899cce519f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ed8581-628f-42ce-8ad0-8899cce519f2-istiod-ca-cert\") pod \"52ed8581-628f-42ce-8ad0-8899cce519f2\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " Oct 30 22:14:28.529404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.529390 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-workload-certs\") pod \"52ed8581-628f-42ce-8ad0-8899cce519f2\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " Oct 30 22:14:28.529520 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.529476 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ed8581-628f-42ce-8ad0-8899cce519f2-kube-api-access-gl5zf\" (UniqueName: \"kubernetes.io/projected/52ed8581-628f-42ce-8ad0-8899cce519f2-kube-api-access-gl5zf\") pod \"52ed8581-628f-42ce-8ad0-8899cce519f2\" (UID: \"52ed8581-628f-42ce-8ad0-8899cce519f2\") " Oct 30 22:14:28.530360 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.530332 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.530477 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.530363 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z85qj\" (UniqueName: \"kubernetes.io/projected/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-kube-api-access-z85qj\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.530477 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.530379 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.530477 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.530392 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.530477 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.530405 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bcc09d-5fcd-45d6-86e6-83bd39d98370-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.535380 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.535105 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-workload-socket" pod "52ed8581-628f-42ce-8ad0-8899cce519f2" (UID: "52ed8581-628f-42ce-8ad0-8899cce519f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:28.537213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.535750 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-workload-certs" pod "52ed8581-628f-42ce-8ad0-8899cce519f2" (UID: "52ed8581-628f-42ce-8ad0-8899cce519f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:28.537213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.536009 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ed8581-628f-42ce-8ad0-8899cce519f2-istiod-ca-cert" pod "52ed8581-628f-42ce-8ad0-8899cce519f2" (UID: "52ed8581-628f-42ce-8ad0-8899cce519f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:28.537213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.536700 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-credential-socket" pod "52ed8581-628f-42ce-8ad0-8899cce519f2" (UID: "52ed8581-628f-42ce-8ad0-8899cce519f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:28.537213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.537183 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-data" pod "52ed8581-628f-42ce-8ad0-8899cce519f2" (UID: "52ed8581-628f-42ce-8ad0-8899cce519f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:28.548806 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.548767 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ed8581-628f-42ce-8ad0-8899cce519f2-kube-api-access-gl5zf" pod "52ed8581-628f-42ce-8ad0-8899cce519f2" (UID: "52ed8581-628f-42ce-8ad0-8899cce519f2"). InnerVolumeSpecName "kube-api-access-gl5zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:28.549819 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.549792 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-podinfo" pod "52ed8581-628f-42ce-8ad0-8899cce519f2" (UID: "52ed8581-628f-42ce-8ad0-8899cce519f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:28.550288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.550251 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-envoy" pod "52ed8581-628f-42ce-8ad0-8899cce519f2" (UID: "52ed8581-628f-42ce-8ad0-8899cce519f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:28.550798 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.550774 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-token" pod "52ed8581-628f-42ce-8ad0-8899cce519f2" (UID: "52ed8581-628f-42ce-8ad0-8899cce519f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:28.591649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.591598 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:14:28.632945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.632888 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.632945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.632922 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.632945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.632941 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.633140 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.632955 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.633140 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.632969 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ed8581-628f-42ce-8ad0-8899cce519f2-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.633140 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.632982 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ed8581-628f-42ce-8ad0-8899cce519f2-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.633140 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.632995 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl5zf\" (UniqueName: \"kubernetes.io/projected/52ed8581-628f-42ce-8ad0-8899cce519f2-kube-api-access-gl5zf\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.633140 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.633012 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.633140 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.633024 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ed8581-628f-42ce-8ad0-8899cce519f2-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.733813 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.733676 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-data\") pod \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " Oct 30 22:14:28.734412 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.734381 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-workload-certs\") pod \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " Oct 30 22:14:28.734535 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.734467 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-envoy\") pod \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " Oct 30 22:14:28.734595 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.734538 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-kube-api-access-rccqw\" (UniqueName: \"kubernetes.io/projected/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-kube-api-access-rccqw\") pod \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " Oct 30 22:14:28.734595 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.734587 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istiod-ca-cert\") pod \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " Oct 30 22:14:28.734763 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.734664 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-workload-socket\") pod \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " Oct 30 22:14:28.734763 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.734735 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-token\" (UniqueName: \"kubernetes.io/projected/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-token\") pod \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " Oct 30 22:14:28.734866 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.734834 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-credential-socket\") pod \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " Oct 30 22:14:28.734946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.734930 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-podinfo\") pod \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\" (UID: \"5fabaab8-b9bc-4346-946c-a7bfd8338dc8\") " Oct 30 22:14:28.736320 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.736201 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istiod-ca-cert" pod "5fabaab8-b9bc-4346-946c-a7bfd8338dc8" (UID: "5fabaab8-b9bc-4346-946c-a7bfd8338dc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:28.736774 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.736712 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-workload-certs" pod "5fabaab8-b9bc-4346-946c-a7bfd8338dc8" (UID: "5fabaab8-b9bc-4346-946c-a7bfd8338dc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:28.736876 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.736785 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-credential-socket" pod "5fabaab8-b9bc-4346-946c-a7bfd8338dc8" (UID: "5fabaab8-b9bc-4346-946c-a7bfd8338dc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:28.737145 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.737053 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.737145 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.737113 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.737145 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.737111 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-workload-socket" pod "5fabaab8-b9bc-4346-946c-a7bfd8338dc8" (UID: "5fabaab8-b9bc-4346-946c-a7bfd8338dc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:28.737145 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.737127 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.737646 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.737605 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-data" pod "5fabaab8-b9bc-4346-946c-a7bfd8338dc8" (UID: "5fabaab8-b9bc-4346-946c-a7bfd8338dc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:28.739234 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.739205 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-token" pod "5fabaab8-b9bc-4346-946c-a7bfd8338dc8" (UID: "5fabaab8-b9bc-4346-946c-a7bfd8338dc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:28.740028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.739990 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-kube-api-access-rccqw" pod "5fabaab8-b9bc-4346-946c-a7bfd8338dc8" (UID: "5fabaab8-b9bc-4346-946c-a7bfd8338dc8"). InnerVolumeSpecName "kube-api-access-rccqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:28.740250 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.740215 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-podinfo" pod "5fabaab8-b9bc-4346-946c-a7bfd8338dc8" (UID: "5fabaab8-b9bc-4346-946c-a7bfd8338dc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:28.742842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.742822 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-envoy" pod "5fabaab8-b9bc-4346-946c-a7bfd8338dc8" (UID: "5fabaab8-b9bc-4346-946c-a7bfd8338dc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:28.838602 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.838576 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.838602 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.838604 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rccqw\" (UniqueName: \"kubernetes.io/projected/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-kube-api-access-rccqw\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.838786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.838632 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.838786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.838643 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.838786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.838651 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:28.838786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:28.838658 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fabaab8-b9bc-4346-946c-a7bfd8338dc8-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:29.077402 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:29.077369 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6"] Oct 30 22:14:29.090534 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:29.090509 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-934ee986-tmjx6"] Oct 30 22:14:29.308588 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:29.308557 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" Oct 30 22:14:29.309157 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:29.308578 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx" event={"ID":"52ed8581-628f-42ce-8ad0-8899cce519f2","Type":"ContainerDied","Data":"fb635dde3caae9992fd0e97e939a8e08e1a4d012d234cdab4c1b2ad269b57286"} Oct 30 22:14:29.309382 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:29.309219 4335 scope.go:122] "RemoveContainer" containerID="5c7b0dedde604dbea078fdff940dec42807d6563d5c5ceb3be0949deb9a90722" Oct 30 22:14:29.313391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:29.310450 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" event={"ID":"b6bcc09d-5fcd-45d6-86e6-83bd39d98370","Type":"ContainerDied","Data":"272c0def75a2f5ec3f7429e84b76c29fa1f9caeb48aae51f6a7aa59d87b08107"} Oct 30 22:14:29.313391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:29.310525 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb" Oct 30 22:14:29.313932 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:29.313652 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" event={"ID":"5fabaab8-b9bc-4346-946c-a7bfd8338dc8","Type":"ContainerDied","Data":"1dbfeca05c5d175b1df779a595983f95757772e97411dc840436f0ea95bd4190"} Oct 30 22:14:29.313932 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:29.313776 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9" Oct 30 22:14:29.327815 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:29.327791 4335 scope.go:122] "RemoveContainer" containerID="d656f97c9f315710c3013dda4fd5c5a9b59f3478578c3757eb9502ebda00a912" Oct 30 22:14:29.340881 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:29.340859 4335 scope.go:122] "RemoveContainer" containerID="74f640cfe3f8ceb96096b43b1f2f969790531548293524fed6c8c56ed532bacf" Oct 30 22:14:29.519339 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:29.519305 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb"] Oct 30 22:14:29.564131 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:29.564096 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-62hlb"] Oct 30 22:14:29.831935 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:29.831899 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9"] Oct 30 22:14:29.870257 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:29.870214 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-q56b9"] Oct 30 22:14:29.899636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:29.897660 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx"] Oct 30 22:14:29.902604 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:29.902576 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-aa8c82d5-openshift-default-7949b48845-8kqfx"] Oct 30 22:14:30.558820 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:30.558781 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-qt88j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:14:30.559463 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:30.559414 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" podUID="476ce668-42e8-41b9-aea8-9727298c9597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:30.713649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:30.713072 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:30.714948 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:30.714803 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:30.719054 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:30.719017 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:14:31.064392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:31.064355 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ed8581-628f-42ce-8ad0-8899cce519f2" path="/var/lib/kubelet/pods/52ed8581-628f-42ce-8ad0-8899cce519f2/volumes" Oct 30 22:14:31.065196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:31.065172 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fabaab8-b9bc-4346-946c-a7bfd8338dc8" path="/var/lib/kubelet/pods/5fabaab8-b9bc-4346-946c-a7bfd8338dc8/volumes" Oct 30 22:14:31.065905 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:31.065886 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6bcc09d-5fcd-45d6-86e6-83bd39d98370" path="/var/lib/kubelet/pods/b6bcc09d-5fcd-45d6-86e6-83bd39d98370/volumes" Oct 30 22:14:31.066993 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:31.066971 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de62402c-594f-4a89-910c-434540713997" path="/var/lib/kubelet/pods/de62402c-594f-4a89-910c-434540713997/volumes" Oct 30 22:14:38.298411 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:38.298383 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz" Oct 30 22:14:39.591962 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.591931 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks"] Oct 30 22:14:39.592670 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.592641 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5fabaab8-b9bc-4346-946c-a7bfd8338dc8" containerName="istio-proxy" Oct 30 22:14:39.592670 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.592666 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fabaab8-b9bc-4346-946c-a7bfd8338dc8" containerName="istio-proxy" Oct 30 22:14:39.592782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.592690 4335 cpu_manager.go:414] "containerMap: removing container" podUID="52ed8581-628f-42ce-8ad0-8899cce519f2" containerName="istio-proxy" Oct 30 22:14:39.592782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.592696 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ed8581-628f-42ce-8ad0-8899cce519f2" containerName="istio-proxy" Oct 30 22:14:39.592782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.592711 4335 cpu_manager.go:414] "containerMap: removing container" podUID="52ed8581-628f-42ce-8ad0-8899cce519f2" containerName="istio-proxy" Oct 30 22:14:39.592782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.592716 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ed8581-628f-42ce-8ad0-8899cce519f2" containerName="istio-proxy" Oct 30 22:14:39.592782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.592724 4335 cpu_manager.go:414] "containerMap: removing container" podUID="b6bcc09d-5fcd-45d6-86e6-83bd39d98370" containerName="istio-proxy" Oct 30 22:14:39.592782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.592729 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bcc09d-5fcd-45d6-86e6-83bd39d98370" containerName="istio-proxy" Oct 30 22:14:39.592970 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.592879 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ed8581-628f-42ce-8ad0-8899cce519f2" containerName="istio-proxy" Oct 30 22:14:39.592970 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.592887 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bcc09d-5fcd-45d6-86e6-83bd39d98370" containerName="istio-proxy" Oct 30 22:14:39.592970 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.592901 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fabaab8-b9bc-4346-946c-a7bfd8338dc8" containerName="istio-proxy" Oct 30 22:14:39.598296 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.598276 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks" Oct 30 22:14:39.667639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.665327 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks"] Oct 30 22:14:39.672252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.669816 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6d6940-a2e0-43fa-86af-0f84f6a22a39-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks\" (UID: \"9a6d6940-a2e0-43fa-86af-0f84f6a22a39\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks" Oct 30 22:14:39.672252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.669958 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmq6x\" (UniqueName: \"kubernetes.io/projected/9a6d6940-a2e0-43fa-86af-0f84f6a22a39-kube-api-access-fmq6x\") pod \"payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks\" (UID: \"9a6d6940-a2e0-43fa-86af-0f84f6a22a39\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks" Oct 30 22:14:39.723640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.723555 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3b355441\"" Oct 30 22:14:39.723840 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.723685 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3b355441-dockercfg-dsftb\"" Oct 30 22:14:39.737645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.734041 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m"] Oct 30 22:14:39.737645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.736694 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ed8581-628f-42ce-8ad0-8899cce519f2" containerName="istio-proxy" Oct 30 22:14:39.744462 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.741957 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m" Oct 30 22:14:39.794652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.793644 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m"] Oct 30 22:14:39.810777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.798584 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6d6940-a2e0-43fa-86af-0f84f6a22a39-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks\" (UID: \"9a6d6940-a2e0-43fa-86af-0f84f6a22a39\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks" Oct 30 22:14:39.810777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.798723 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmq6x\" (UniqueName: \"kubernetes.io/projected/9a6d6940-a2e0-43fa-86af-0f84f6a22a39-kube-api-access-fmq6x\") pod \"payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks\" (UID: \"9a6d6940-a2e0-43fa-86af-0f84f6a22a39\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks" Oct 30 22:14:39.810777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.799944 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6d6940-a2e0-43fa-86af-0f84f6a22a39-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks\" (UID: \"9a6d6940-a2e0-43fa-86af-0f84f6a22a39\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks" Oct 30 22:14:39.865707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.861257 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmq6x\" (UniqueName: \"kubernetes.io/projected/9a6d6940-a2e0-43fa-86af-0f84f6a22a39-kube-api-access-fmq6x\") pod \"payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks\" (UID: \"9a6d6940-a2e0-43fa-86af-0f84f6a22a39\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks" Oct 30 22:14:39.909222 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.909190 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq7tq\" (UniqueName: \"kubernetes.io/projected/67957741-7440-4dd3-9ddc-10dc3963b5f3-kube-api-access-bq7tq\") pod \"payload-processing-e2e-mt-3b355441-57446bc65-j5t4m\" (UID: \"67957741-7440-4dd3-9ddc-10dc3963b5f3\") " pod="openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m" Oct 30 22:14:39.909429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.909417 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67957741-7440-4dd3-9ddc-10dc3963b5f3-plugins-config-volume\") pod \"payload-processing-e2e-mt-3b355441-57446bc65-j5t4m\" (UID: \"67957741-7440-4dd3-9ddc-10dc3963b5f3\") " pod="openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m" Oct 30 22:14:39.932063 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:39.932037 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks" Oct 30 22:14:40.027639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:40.023461 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq7tq\" (UniqueName: \"kubernetes.io/projected/67957741-7440-4dd3-9ddc-10dc3963b5f3-kube-api-access-bq7tq\") pod \"payload-processing-e2e-mt-3b355441-57446bc65-j5t4m\" (UID: \"67957741-7440-4dd3-9ddc-10dc3963b5f3\") " pod="openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m" Oct 30 22:14:40.027639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:40.023513 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67957741-7440-4dd3-9ddc-10dc3963b5f3-plugins-config-volume\") pod \"payload-processing-e2e-mt-3b355441-57446bc65-j5t4m\" (UID: \"67957741-7440-4dd3-9ddc-10dc3963b5f3\") " pod="openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m" Oct 30 22:14:40.027639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:40.024760 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67957741-7440-4dd3-9ddc-10dc3963b5f3-plugins-config-volume\") pod \"payload-processing-e2e-mt-3b355441-57446bc65-j5t4m\" (UID: \"67957741-7440-4dd3-9ddc-10dc3963b5f3\") " pod="openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m" Oct 30 22:14:40.195720 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:40.162091 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq7tq\" (UniqueName: \"kubernetes.io/projected/67957741-7440-4dd3-9ddc-10dc3963b5f3-kube-api-access-bq7tq\") pod \"payload-processing-e2e-mt-3b355441-57446bc65-j5t4m\" (UID: \"67957741-7440-4dd3-9ddc-10dc3963b5f3\") " pod="openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m" Oct 30 22:14:40.429292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:40.429221 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m" Oct 30 22:14:40.548879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:40.548851 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks"] Oct 30 22:14:40.817213 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:14:40.817181 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67957741_7440_4dd3_9ddc_10dc3963b5f3.slice/crio-bc2cf9af7058b7e712722adaec3a7f4ec2bddce856ea3178344f4e2756210850 WatchSource:0}: Error finding container bc2cf9af7058b7e712722adaec3a7f4ec2bddce856ea3178344f4e2756210850: Status 404 returned error can't find the container with id bc2cf9af7058b7e712722adaec3a7f4ec2bddce856ea3178344f4e2756210850 Oct 30 22:14:40.817213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:40.817197 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m"] Oct 30 22:14:41.502693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:41.502648 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m" event={"ID":"67957741-7440-4dd3-9ddc-10dc3963b5f3","Type":"ContainerStarted","Data":"e5d8ff981f796bd84046c9ae276f5699fbe7be6de46a749c66bd6e6d1991f4df"} Oct 30 22:14:41.502875 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:41.502717 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m" event={"ID":"67957741-7440-4dd3-9ddc-10dc3963b5f3","Type":"ContainerStarted","Data":"bc2cf9af7058b7e712722adaec3a7f4ec2bddce856ea3178344f4e2756210850"} Oct 30 22:14:41.502875 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:41.502748 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m" Oct 30 22:14:41.504800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:41.504761 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks" event={"ID":"9a6d6940-a2e0-43fa-86af-0f84f6a22a39","Type":"ContainerStarted","Data":"599b9df090b1d30d0a9037a0662cd44a2624bc7b7a33248fbb6e2b700f9a3097"} Oct 30 22:14:41.504936 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:41.504812 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks" event={"ID":"9a6d6940-a2e0-43fa-86af-0f84f6a22a39","Type":"ContainerStarted","Data":"d68bd0cff3332678942fb748904ab96698ba79d7f2ea17fb587a1b77ce58b8ac"} Oct 30 22:14:41.504936 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:41.504891 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks" Oct 30 22:14:41.617156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:41.617097 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m" podStartSLOduration=2.617079909 podStartE2EDuration="2.617079909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:41.616567103 +0000 UTC m=+1814.716998835" watchObservedRunningTime="2026-10-30 22:14:41.617079909 +0000 UTC m=+1814.717511642" Oct 30 22:14:41.742831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:41.742763 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks" podStartSLOduration=2.742743244 podStartE2EDuration="2.742743244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:41.731571885 +0000 UTC m=+1814.832003619" watchObservedRunningTime="2026-10-30 22:14:41.742743244 +0000 UTC m=+1814.843174992" Oct 30 22:14:45.061537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:45.061507 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5"] Oct 30 22:14:45.101511 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:45.101483 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eeb3b5f-mkvb5"] Oct 30 22:14:46.050816 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:46.050783 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk"] Oct 30 22:14:46.099580 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:46.099553 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d92705e-mghsk"] Oct 30 22:14:47.059805 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:47.059771 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41189525-2831-42f8-bcec-8426752c71e8" path="/var/lib/kubelet/pods/41189525-2831-42f8-bcec-8426752c71e8/volumes" Oct 30 22:14:47.060259 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:47.060243 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f98131f4-7440-4740-b286-3dea16a9f939" path="/var/lib/kubelet/pods/f98131f4-7440-4740-b286-3dea16a9f939/volumes" Oct 30 22:14:51.062149 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:51.062115 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq"] Oct 30 22:14:51.104073 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:51.104047 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54ced5c2-277wq"] Oct 30 22:14:52.096665 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:52.096579 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s"] Oct 30 22:14:52.145776 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:52.145749 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4031e7a9-k797s"] Oct 30 22:14:52.518331 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:52.518266 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m" Oct 30 22:14:52.518473 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:52.518412 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks" Oct 30 22:14:53.060273 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:53.060235 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49045d81-6fce-40bf-8adc-8bffaaeefbc6" path="/var/lib/kubelet/pods/49045d81-6fce-40bf-8adc-8bffaaeefbc6/volumes" Oct 30 22:14:53.060785 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:53.060767 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d45ce109-ed30-459e-954f-452d1ab9e7c4" path="/var/lib/kubelet/pods/d45ce109-ed30-459e-954f-452d1ab9e7c4/volumes" Oct 30 22:14:54.714966 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:54.714937 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22"] Oct 30 22:14:54.730471 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:54.730433 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22"] Oct 30 22:14:54.730642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:54.730586 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22" Oct 30 22:14:54.782783 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:54.782757 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-9a96b4\"" Oct 30 22:14:54.798169 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:54.798132 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6c6f\" (UniqueName: \"kubernetes.io/projected/171de552-11b5-4865-b021-92ae8807842d-kube-api-access-g6c6f\") pod \"payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22\" (UID: \"171de552-11b5-4865-b021-92ae8807842d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22" Oct 30 22:14:54.798355 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:54.798335 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171de552-11b5-4865-b021-92ae8807842d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22\" (UID: \"171de552-11b5-4865-b021-92ae8807842d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22" Oct 30 22:14:54.832018 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:54.831993 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-9a96b4-dockercfg-4qpbx\"" Oct 30 22:14:54.900165 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:54.900135 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171de552-11b5-4865-b021-92ae8807842d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22\" (UID: \"171de552-11b5-4865-b021-92ae8807842d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22" Oct 30 22:14:54.900316 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:54.900285 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6c6f\" (UniqueName: \"kubernetes.io/projected/171de552-11b5-4865-b021-92ae8807842d-kube-api-access-g6c6f\") pod \"payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22\" (UID: \"171de552-11b5-4865-b021-92ae8807842d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22" Oct 30 22:14:54.900885 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:54.900866 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171de552-11b5-4865-b021-92ae8807842d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22\" (UID: \"171de552-11b5-4865-b021-92ae8807842d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22" Oct 30 22:14:54.961232 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:54.961206 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6c6f\" (UniqueName: \"kubernetes.io/projected/171de552-11b5-4865-b021-92ae8807842d-kube-api-access-g6c6f\") pod \"payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22\" (UID: \"171de552-11b5-4865-b021-92ae8807842d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22" Oct 30 22:14:55.048146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:55.048071 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22" Oct 30 22:14:55.082328 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:55.082304 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7"] Oct 30 22:14:55.082441 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:55.082419 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98daec9f-hfcn7"] Oct 30 22:14:55.322131 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:14:55.322098 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:14:55.468196 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:14:55.468172 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171de552_11b5_4865_b021_92ae8807842d.slice/crio-0ab5115a3fd5487eed2fb4d638d121536786341986a5c7572908ffb275bceeb7 WatchSource:0}: Error finding container 0ab5115a3fd5487eed2fb4d638d121536786341986a5c7572908ffb275bceeb7: Status 404 returned error can't find the container with id 0ab5115a3fd5487eed2fb4d638d121536786341986a5c7572908ffb275bceeb7 Oct 30 22:14:55.468835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:55.468810 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22"] Oct 30 22:14:55.728711 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:55.728625 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22" event={"ID":"171de552-11b5-4865-b021-92ae8807842d","Type":"ContainerStarted","Data":"8b9fdf15aa4d9d40b94cea0b9a88971a8008b45959e1a793b24246e542d85cbf"} Oct 30 22:14:55.728711 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:55.728699 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22" event={"ID":"171de552-11b5-4865-b021-92ae8807842d","Type":"ContainerStarted","Data":"0ab5115a3fd5487eed2fb4d638d121536786341986a5c7572908ffb275bceeb7"} Oct 30 22:14:55.729091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:55.728734 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22" Oct 30 22:14:55.848313 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:55.848261 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22" podStartSLOduration=1.848249097 podStartE2EDuration="1.848249097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:55.843481747 +0000 UTC m=+1828.943913479" watchObservedRunningTime="2026-10-30 22:14:55.848249097 +0000 UTC m=+1828.948680824" Oct 30 22:14:57.059786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:14:57.059756 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a255089b-e277-48a1-90b0-043db61a9443" path="/var/lib/kubelet/pods/a255089b-e277-48a1-90b0-043db61a9443/volumes" Oct 30 22:15:00.052284 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.052249 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv"] Oct 30 22:15:00.100340 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.100308 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a82af8e-jkhkv"] Oct 30 22:15:00.190478 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.190441 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc"] Oct 30 22:15:00.197750 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.197714 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf"] Oct 30 22:15:00.197890 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.197841 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" Oct 30 22:15:00.203128 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.203108 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx"] Oct 30 22:15:00.203320 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.203215 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" Oct 30 22:15:00.208386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.208364 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" Oct 30 22:15:00.232561 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.232538 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq"] Oct 30 22:15:00.236994 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.236976 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" Oct 30 22:15:00.237202 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.237184 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh"] Oct 30 22:15:00.242255 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.242238 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" Oct 30 22:15:00.250318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.250293 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:15:00.250462 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.250423 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:15:00.255403 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.255386 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc"] Oct 30 22:15:00.262314 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.262290 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf"] Oct 30 22:15:00.276132 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.276101 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:15:00.296289 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.296261 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx"] Oct 30 22:15:00.309114 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.309015 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh"] Oct 30 22:15:00.360979 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.360948 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq"] Oct 30 22:15:00.374363 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.374338 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2p68\" (UniqueName: \"kubernetes.io/projected/5ff5482f-138e-4f1b-afad-d4dc9dc99ebf-kube-api-access-x2p68\") pod \"maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf\" (UID: \"5ff5482f-138e-4f1b-afad-d4dc9dc99ebf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" Oct 30 22:15:00.374491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.374398 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2646e7c5-c250-40b0-af13-d62e8b17aff6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh\" (UID: \"2646e7c5-c250-40b0-af13-d62e8b17aff6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" Oct 30 22:15:00.374553 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.374529 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7j5n\" (UniqueName: \"kubernetes.io/projected/f3a89707-003c-4807-95d3-0bb14607ffdd-kube-api-access-x7j5n\") pod \"maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx\" (UID: \"f3a89707-003c-4807-95d3-0bb14607ffdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" Oct 30 22:15:00.374712 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.374692 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqzl9\" (UniqueName: \"kubernetes.io/projected/0422577b-0c47-4546-80e2-518a849c135b-kube-api-access-rqzl9\") pod \"maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc\" (UID: \"0422577b-0c47-4546-80e2-518a849c135b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" Oct 30 22:15:00.374775 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.374733 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljgrh\" (UniqueName: \"kubernetes.io/projected/2646e7c5-c250-40b0-af13-d62e8b17aff6-kube-api-access-ljgrh\") pod \"maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh\" (UID: \"2646e7c5-c250-40b0-af13-d62e8b17aff6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" Oct 30 22:15:00.374775 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.374757 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff5482f-138e-4f1b-afad-d4dc9dc99ebf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf\" (UID: \"5ff5482f-138e-4f1b-afad-d4dc9dc99ebf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" Oct 30 22:15:00.374971 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.374953 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a89707-003c-4807-95d3-0bb14607ffdd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx\" (UID: \"f3a89707-003c-4807-95d3-0bb14607ffdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" Oct 30 22:15:00.375034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.375004 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0422577b-0c47-4546-80e2-518a849c135b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc\" (UID: \"0422577b-0c47-4546-80e2-518a849c135b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" Oct 30 22:15:00.375091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.375075 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b5e2a6-85d0-4149-9c26-d65096c5a496-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq\" (UID: \"f2b5e2a6-85d0-4149-9c26-d65096c5a496\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" Oct 30 22:15:00.375166 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.375151 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvvm7\" (UniqueName: \"kubernetes.io/projected/f2b5e2a6-85d0-4149-9c26-d65096c5a496-kube-api-access-kvvm7\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq\" (UID: \"f2b5e2a6-85d0-4149-9c26-d65096c5a496\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" Oct 30 22:15:00.476171 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.476143 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqzl9\" (UniqueName: \"kubernetes.io/projected/0422577b-0c47-4546-80e2-518a849c135b-kube-api-access-rqzl9\") pod \"maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc\" (UID: \"0422577b-0c47-4546-80e2-518a849c135b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" Oct 30 22:15:00.476289 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.476185 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljgrh\" (UniqueName: \"kubernetes.io/projected/2646e7c5-c250-40b0-af13-d62e8b17aff6-kube-api-access-ljgrh\") pod \"maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh\" (UID: \"2646e7c5-c250-40b0-af13-d62e8b17aff6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" Oct 30 22:15:00.476289 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.476207 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff5482f-138e-4f1b-afad-d4dc9dc99ebf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf\" (UID: \"5ff5482f-138e-4f1b-afad-d4dc9dc99ebf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" Oct 30 22:15:00.476381 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.476294 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a89707-003c-4807-95d3-0bb14607ffdd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx\" (UID: \"f3a89707-003c-4807-95d3-0bb14607ffdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" Oct 30 22:15:00.476440 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.476418 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0422577b-0c47-4546-80e2-518a849c135b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc\" (UID: \"0422577b-0c47-4546-80e2-518a849c135b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" Oct 30 22:15:00.476515 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.476485 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b5e2a6-85d0-4149-9c26-d65096c5a496-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq\" (UID: \"f2b5e2a6-85d0-4149-9c26-d65096c5a496\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" Oct 30 22:15:00.476598 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.476584 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvvm7\" (UniqueName: \"kubernetes.io/projected/f2b5e2a6-85d0-4149-9c26-d65096c5a496-kube-api-access-kvvm7\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq\" (UID: \"f2b5e2a6-85d0-4149-9c26-d65096c5a496\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" Oct 30 22:15:00.476691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.476674 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2p68\" (UniqueName: \"kubernetes.io/projected/5ff5482f-138e-4f1b-afad-d4dc9dc99ebf-kube-api-access-x2p68\") pod \"maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf\" (UID: \"5ff5482f-138e-4f1b-afad-d4dc9dc99ebf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" Oct 30 22:15:00.476751 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.476737 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2646e7c5-c250-40b0-af13-d62e8b17aff6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh\" (UID: \"2646e7c5-c250-40b0-af13-d62e8b17aff6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" Oct 30 22:15:00.476810 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.476772 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7j5n\" (UniqueName: \"kubernetes.io/projected/f3a89707-003c-4807-95d3-0bb14607ffdd-kube-api-access-x7j5n\") pod \"maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx\" (UID: \"f3a89707-003c-4807-95d3-0bb14607ffdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" Oct 30 22:15:00.477161 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.477119 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a89707-003c-4807-95d3-0bb14607ffdd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx\" (UID: \"f3a89707-003c-4807-95d3-0bb14607ffdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" Oct 30 22:15:00.477161 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.477138 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff5482f-138e-4f1b-afad-d4dc9dc99ebf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf\" (UID: \"5ff5482f-138e-4f1b-afad-d4dc9dc99ebf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" Oct 30 22:15:00.477337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.477210 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b5e2a6-85d0-4149-9c26-d65096c5a496-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq\" (UID: \"f2b5e2a6-85d0-4149-9c26-d65096c5a496\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" Oct 30 22:15:00.477337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.477229 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0422577b-0c47-4546-80e2-518a849c135b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc\" (UID: \"0422577b-0c47-4546-80e2-518a849c135b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" Oct 30 22:15:00.477569 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.477540 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2646e7c5-c250-40b0-af13-d62e8b17aff6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh\" (UID: \"2646e7c5-c250-40b0-af13-d62e8b17aff6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" Oct 30 22:15:00.538204 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.538175 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvvm7\" (UniqueName: \"kubernetes.io/projected/f2b5e2a6-85d0-4149-9c26-d65096c5a496-kube-api-access-kvvm7\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq\" (UID: \"f2b5e2a6-85d0-4149-9c26-d65096c5a496\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" Oct 30 22:15:00.539495 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.539474 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7j5n\" (UniqueName: \"kubernetes.io/projected/f3a89707-003c-4807-95d3-0bb14607ffdd-kube-api-access-x7j5n\") pod \"maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx\" (UID: \"f3a89707-003c-4807-95d3-0bb14607ffdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" Oct 30 22:15:00.539904 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.539881 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljgrh\" (UniqueName: \"kubernetes.io/projected/2646e7c5-c250-40b0-af13-d62e8b17aff6-kube-api-access-ljgrh\") pod \"maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh\" (UID: \"2646e7c5-c250-40b0-af13-d62e8b17aff6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" Oct 30 22:15:00.540004 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.539955 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqzl9\" (UniqueName: \"kubernetes.io/projected/0422577b-0c47-4546-80e2-518a849c135b-kube-api-access-rqzl9\") pod \"maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc\" (UID: \"0422577b-0c47-4546-80e2-518a849c135b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" Oct 30 22:15:00.540392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.540370 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2p68\" (UniqueName: \"kubernetes.io/projected/5ff5482f-138e-4f1b-afad-d4dc9dc99ebf-kube-api-access-x2p68\") pod \"maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf\" (UID: \"5ff5482f-138e-4f1b-afad-d4dc9dc99ebf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" Oct 30 22:15:00.555827 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.555806 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" Oct 30 22:15:00.562078 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.562057 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" Oct 30 22:15:00.815554 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.815483 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" Oct 30 22:15:00.822896 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.822868 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" Oct 30 22:15:00.831561 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.831539 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" Oct 30 22:15:00.961460 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:00.961425 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq"] Oct 30 22:15:01.002923 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:01.002898 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh"] Oct 30 22:15:01.065815 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:01.065746 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24d36705-49a0-44ac-b0fe-11f8bd39d06c" path="/var/lib/kubelet/pods/24d36705-49a0-44ac-b0fe-11f8bd39d06c/volumes" Oct 30 22:15:01.233546 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:15:01.233516 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff5482f_138e_4f1b_afad_d4dc9dc99ebf.slice/crio-5c95ac5ca9286034c19ef76d9b96908c16c4676b49430a8bef8dd97dc2a4e7c2 WatchSource:0}: Error finding container 5c95ac5ca9286034c19ef76d9b96908c16c4676b49430a8bef8dd97dc2a4e7c2: Status 404 returned error can't find the container with id 5c95ac5ca9286034c19ef76d9b96908c16c4676b49430a8bef8dd97dc2a4e7c2 Oct 30 22:15:01.238335 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:01.238316 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf"] Oct 30 22:15:01.256554 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:15:01.256449 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0422577b_0c47_4546_80e2_518a849c135b.slice/crio-f32a0decd536eae95c7ba6988cde763278f1bc0dc7e458f705f9b2e0f5cb9bdc WatchSource:0}: Error finding container f32a0decd536eae95c7ba6988cde763278f1bc0dc7e458f705f9b2e0f5cb9bdc: Status 404 returned error can't find the container with id f32a0decd536eae95c7ba6988cde763278f1bc0dc7e458f705f9b2e0f5cb9bdc Oct 30 22:15:01.258770 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:01.258746 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc"] Oct 30 22:15:01.271726 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:15:01.271700 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a89707_003c_4807_95d3_0bb14607ffdd.slice/crio-582a59109d5385148bb51c468b2015ee8fc5bd2985b3f12348cb3f7958975af6 WatchSource:0}: Error finding container 582a59109d5385148bb51c468b2015ee8fc5bd2985b3f12348cb3f7958975af6: Status 404 returned error can't find the container with id 582a59109d5385148bb51c468b2015ee8fc5bd2985b3f12348cb3f7958975af6 Oct 30 22:15:01.272162 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:01.272139 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx"] Oct 30 22:15:01.797128 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:01.797093 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" event={"ID":"0422577b-0c47-4546-80e2-518a849c135b","Type":"ContainerStarted","Data":"d00a92ea0ea03ad5d92b4aad2d169a032b952c365e3b1c87e87ba6cb4ea199b5"} Oct 30 22:15:01.797289 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:01.797157 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" event={"ID":"0422577b-0c47-4546-80e2-518a849c135b","Type":"ContainerStarted","Data":"f32a0decd536eae95c7ba6988cde763278f1bc0dc7e458f705f9b2e0f5cb9bdc"} Oct 30 22:15:01.801295 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:01.801264 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" event={"ID":"5ff5482f-138e-4f1b-afad-d4dc9dc99ebf","Type":"ContainerStarted","Data":"5946574c585819cfb8f2d185b02ee88aaca0c2d6130efa8308018c1029e03c07"} Oct 30 22:15:01.801409 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:01.801321 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" event={"ID":"5ff5482f-138e-4f1b-afad-d4dc9dc99ebf","Type":"ContainerStarted","Data":"5c95ac5ca9286034c19ef76d9b96908c16c4676b49430a8bef8dd97dc2a4e7c2"} Oct 30 22:15:01.803337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:01.803311 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" event={"ID":"f3a89707-003c-4807-95d3-0bb14607ffdd","Type":"ContainerStarted","Data":"89fbd8f64e4c2956066523547ebdef1c6281b984d9b0b69eddb94d03287e8b4b"} Oct 30 22:15:01.803437 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:01.803357 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" event={"ID":"f3a89707-003c-4807-95d3-0bb14607ffdd","Type":"ContainerStarted","Data":"582a59109d5385148bb51c468b2015ee8fc5bd2985b3f12348cb3f7958975af6"} Oct 30 22:15:01.804916 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:01.804892 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" event={"ID":"2646e7c5-c250-40b0-af13-d62e8b17aff6","Type":"ContainerStarted","Data":"fed6ef2d0eebd6562e3c3d4d502f1730a95554ed61748bce51d5b908fa8f83bb"} Oct 30 22:15:01.805001 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:01.804940 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" event={"ID":"2646e7c5-c250-40b0-af13-d62e8b17aff6","Type":"ContainerStarted","Data":"338a2f5a44324cf812399d5717908bdfe705ef0658881c045c6557caf9b95e2a"} Oct 30 22:15:01.806414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:01.806392 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" event={"ID":"f2b5e2a6-85d0-4149-9c26-d65096c5a496","Type":"ContainerStarted","Data":"48ee7d13c3ac00b336cc7b07e2fc9260e7eda6ba3820af16ceea26a58c159b58"} Oct 30 22:15:01.806500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:01.806429 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" event={"ID":"f2b5e2a6-85d0-4149-9c26-d65096c5a496","Type":"ContainerStarted","Data":"f58b2e41ac559be09df91b173b54ba12bb9f446bf33a937ad8f50efc762198e7"} Oct 30 22:15:01.917248 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:01.917141 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" podStartSLOduration=1.917129158 podStartE2EDuration="1.917129158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.911669138 +0000 UTC m=+1835.012100870" watchObservedRunningTime="2026-10-30 22:15:01.917129158 +0000 UTC m=+1835.017560908" Oct 30 22:15:02.032702 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.032643 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" podStartSLOduration=2.032629701 podStartE2EDuration="2.032629701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.026790865 +0000 UTC m=+1835.127222598" watchObservedRunningTime="2026-10-30 22:15:02.032629701 +0000 UTC m=+1835.133061475" Oct 30 22:15:02.145963 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.145919 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" podStartSLOduration=2.145906941 podStartE2EDuration="2.145906941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.14109927 +0000 UTC m=+1835.241531194" watchObservedRunningTime="2026-10-30 22:15:02.145906941 +0000 UTC m=+1835.246338677" Oct 30 22:15:02.259124 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.259081 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" podStartSLOduration=2.259066953 podStartE2EDuration="2.259066953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.255330178 +0000 UTC m=+1835.355761907" watchObservedRunningTime="2026-10-30 22:15:02.259066953 +0000 UTC m=+1835.359498684" Oct 30 22:15:02.373138 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.373094 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" podStartSLOduration=2.373081043 podStartE2EDuration="2.373081043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.368865451 +0000 UTC m=+1835.469297211" watchObservedRunningTime="2026-10-30 22:15:02.373081043 +0000 UTC m=+1835.473512792" Oct 30 22:15:02.819445 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.819412 4335 generic.go:356] "Generic (PLEG): container finished" podID="0422577b-0c47-4546-80e2-518a849c135b" containerID="d00a92ea0ea03ad5d92b4aad2d169a032b952c365e3b1c87e87ba6cb4ea199b5" exitCode=60 Oct 30 22:15:02.819639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.819498 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" event={"ID":"0422577b-0c47-4546-80e2-518a849c135b","Type":"ContainerDied","Data":"d00a92ea0ea03ad5d92b4aad2d169a032b952c365e3b1c87e87ba6cb4ea199b5"} Oct 30 22:15:02.819770 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.819746 4335 scope.go:122] "RemoveContainer" containerID="d00a92ea0ea03ad5d92b4aad2d169a032b952c365e3b1c87e87ba6cb4ea199b5" Oct 30 22:15:02.821544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.821521 4335 generic.go:356] "Generic (PLEG): container finished" podID="5ff5482f-138e-4f1b-afad-d4dc9dc99ebf" containerID="5946574c585819cfb8f2d185b02ee88aaca0c2d6130efa8308018c1029e03c07" exitCode=60 Oct 30 22:15:02.821860 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.821825 4335 scope.go:122] "RemoveContainer" containerID="5946574c585819cfb8f2d185b02ee88aaca0c2d6130efa8308018c1029e03c07" Oct 30 22:15:02.823681 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.823654 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" event={"ID":"5ff5482f-138e-4f1b-afad-d4dc9dc99ebf","Type":"ContainerDied","Data":"5946574c585819cfb8f2d185b02ee88aaca0c2d6130efa8308018c1029e03c07"} Oct 30 22:15:02.828093 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.825580 4335 generic.go:356] "Generic (PLEG): container finished" podID="f3a89707-003c-4807-95d3-0bb14607ffdd" containerID="89fbd8f64e4c2956066523547ebdef1c6281b984d9b0b69eddb94d03287e8b4b" exitCode=60 Oct 30 22:15:02.828093 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.825818 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" event={"ID":"f3a89707-003c-4807-95d3-0bb14607ffdd","Type":"ContainerDied","Data":"89fbd8f64e4c2956066523547ebdef1c6281b984d9b0b69eddb94d03287e8b4b"} Oct 30 22:15:02.828093 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.825963 4335 scope.go:122] "RemoveContainer" containerID="89fbd8f64e4c2956066523547ebdef1c6281b984d9b0b69eddb94d03287e8b4b" Oct 30 22:15:02.828093 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.827748 4335 generic.go:356] "Generic (PLEG): container finished" podID="2646e7c5-c250-40b0-af13-d62e8b17aff6" containerID="fed6ef2d0eebd6562e3c3d4d502f1730a95554ed61748bce51d5b908fa8f83bb" exitCode=60 Oct 30 22:15:02.828093 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.827885 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" event={"ID":"2646e7c5-c250-40b0-af13-d62e8b17aff6","Type":"ContainerDied","Data":"fed6ef2d0eebd6562e3c3d4d502f1730a95554ed61748bce51d5b908fa8f83bb"} Oct 30 22:15:02.828401 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.828234 4335 scope.go:122] "RemoveContainer" containerID="fed6ef2d0eebd6562e3c3d4d502f1730a95554ed61748bce51d5b908fa8f83bb" Oct 30 22:15:02.830835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.830811 4335 generic.go:356] "Generic (PLEG): container finished" podID="f2b5e2a6-85d0-4149-9c26-d65096c5a496" containerID="48ee7d13c3ac00b336cc7b07e2fc9260e7eda6ba3820af16ceea26a58c159b58" exitCode=60 Oct 30 22:15:02.830989 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.830967 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" event={"ID":"f2b5e2a6-85d0-4149-9c26-d65096c5a496","Type":"ContainerDied","Data":"48ee7d13c3ac00b336cc7b07e2fc9260e7eda6ba3820af16ceea26a58c159b58"} Oct 30 22:15:02.831098 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:02.831088 4335 scope.go:122] "RemoveContainer" containerID="48ee7d13c3ac00b336cc7b07e2fc9260e7eda6ba3820af16ceea26a58c159b58" Oct 30 22:15:03.070846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:03.070774 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv"] Oct 30 22:15:03.080144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:03.080120 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3d04b9-z6qtv"] Oct 30 22:15:03.844200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:03.844161 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" event={"ID":"f3a89707-003c-4807-95d3-0bb14607ffdd","Type":"ContainerStarted","Data":"2c20cde09f9b9ca3ee7349ef8a4b10beb153bc7ebb9e0c9d5e22f142ac1e056e"} Oct 30 22:15:03.846493 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:03.846465 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" event={"ID":"2646e7c5-c250-40b0-af13-d62e8b17aff6","Type":"ContainerStarted","Data":"b76908dde95c09dfbf9d42db316e6eb8592e34862bb2efb11f91618887dc7598"} Oct 30 22:15:03.848624 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:03.848587 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" event={"ID":"f2b5e2a6-85d0-4149-9c26-d65096c5a496","Type":"ContainerStarted","Data":"7b4bcd32493f0427907fd36330023b2c0560886e73bd76a1374a4c9f87fc3c1d"} Oct 30 22:15:03.850656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:03.850636 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" event={"ID":"0422577b-0c47-4546-80e2-518a849c135b","Type":"ContainerStarted","Data":"9858df92f32e1b91e04728286c38f7cc076e2132c59f4d4f9e2b730760af64b7"} Oct 30 22:15:03.852656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:03.852591 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" event={"ID":"5ff5482f-138e-4f1b-afad-d4dc9dc99ebf","Type":"ContainerStarted","Data":"ec436a05668041eac1843a8d7d05920e92666717c6ee2eba65e13b7c216d2261"} Oct 30 22:15:04.867676 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.867582 4335 generic.go:356] "Generic (PLEG): container finished" podID="f3a89707-003c-4807-95d3-0bb14607ffdd" containerID="2c20cde09f9b9ca3ee7349ef8a4b10beb153bc7ebb9e0c9d5e22f142ac1e056e" exitCode=60 Oct 30 22:15:04.867676 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.867690 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" event={"ID":"f3a89707-003c-4807-95d3-0bb14607ffdd","Type":"ContainerDied","Data":"2c20cde09f9b9ca3ee7349ef8a4b10beb153bc7ebb9e0c9d5e22f142ac1e056e"} Oct 30 22:15:04.868327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.867743 4335 scope.go:122] "RemoveContainer" containerID="89fbd8f64e4c2956066523547ebdef1c6281b984d9b0b69eddb94d03287e8b4b" Oct 30 22:15:04.868327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.868164 4335 scope.go:122] "RemoveContainer" containerID="2c20cde09f9b9ca3ee7349ef8a4b10beb153bc7ebb9e0c9d5e22f142ac1e056e" Oct 30 22:15:04.868668 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:15:04.868583 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx_odh-ai-gateway-infra(f3a89707-003c-4807-95d3-0bb14607ffdd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" podUID="f3a89707-003c-4807-95d3-0bb14607ffdd" Oct 30 22:15:04.870892 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.870863 4335 generic.go:356] "Generic (PLEG): container finished" podID="2646e7c5-c250-40b0-af13-d62e8b17aff6" containerID="b76908dde95c09dfbf9d42db316e6eb8592e34862bb2efb11f91618887dc7598" exitCode=60 Oct 30 22:15:04.870997 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.870948 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" event={"ID":"2646e7c5-c250-40b0-af13-d62e8b17aff6","Type":"ContainerDied","Data":"b76908dde95c09dfbf9d42db316e6eb8592e34862bb2efb11f91618887dc7598"} Oct 30 22:15:04.871243 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.871226 4335 scope.go:122] "RemoveContainer" containerID="b76908dde95c09dfbf9d42db316e6eb8592e34862bb2efb11f91618887dc7598" Oct 30 22:15:04.871549 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:15:04.871521 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh_odh-ai-gateway-infra(2646e7c5-c250-40b0-af13-d62e8b17aff6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" podUID="2646e7c5-c250-40b0-af13-d62e8b17aff6" Oct 30 22:15:04.874880 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.874859 4335 generic.go:356] "Generic (PLEG): container finished" podID="f2b5e2a6-85d0-4149-9c26-d65096c5a496" containerID="7b4bcd32493f0427907fd36330023b2c0560886e73bd76a1374a4c9f87fc3c1d" exitCode=60 Oct 30 22:15:04.875056 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.874889 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" event={"ID":"f2b5e2a6-85d0-4149-9c26-d65096c5a496","Type":"ContainerDied","Data":"7b4bcd32493f0427907fd36330023b2c0560886e73bd76a1374a4c9f87fc3c1d"} Oct 30 22:15:04.875353 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.875335 4335 scope.go:122] "RemoveContainer" containerID="7b4bcd32493f0427907fd36330023b2c0560886e73bd76a1374a4c9f87fc3c1d" Oct 30 22:15:04.875987 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:15:04.875683 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq_odh-ai-gateway-infra(f2b5e2a6-85d0-4149-9c26-d65096c5a496)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" podUID="f2b5e2a6-85d0-4149-9c26-d65096c5a496" Oct 30 22:15:04.884073 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.884044 4335 generic.go:356] "Generic (PLEG): container finished" podID="0422577b-0c47-4546-80e2-518a849c135b" containerID="9858df92f32e1b91e04728286c38f7cc076e2132c59f4d4f9e2b730760af64b7" exitCode=60 Oct 30 22:15:04.884243 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.884227 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" event={"ID":"0422577b-0c47-4546-80e2-518a849c135b","Type":"ContainerDied","Data":"9858df92f32e1b91e04728286c38f7cc076e2132c59f4d4f9e2b730760af64b7"} Oct 30 22:15:04.884744 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.884720 4335 scope.go:122] "RemoveContainer" containerID="9858df92f32e1b91e04728286c38f7cc076e2132c59f4d4f9e2b730760af64b7" Oct 30 22:15:04.885216 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:15:04.885189 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc_odh-ai-gateway-infra(0422577b-0c47-4546-80e2-518a849c135b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" podUID="0422577b-0c47-4546-80e2-518a849c135b" Oct 30 22:15:04.893135 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.892981 4335 scope.go:122] "RemoveContainer" containerID="fed6ef2d0eebd6562e3c3d4d502f1730a95554ed61748bce51d5b908fa8f83bb" Oct 30 22:15:04.893240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.893138 4335 generic.go:356] "Generic (PLEG): container finished" podID="5ff5482f-138e-4f1b-afad-d4dc9dc99ebf" containerID="ec436a05668041eac1843a8d7d05920e92666717c6ee2eba65e13b7c216d2261" exitCode=60 Oct 30 22:15:04.893811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.893237 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" event={"ID":"5ff5482f-138e-4f1b-afad-d4dc9dc99ebf","Type":"ContainerDied","Data":"ec436a05668041eac1843a8d7d05920e92666717c6ee2eba65e13b7c216d2261"} Oct 30 22:15:04.893811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.893679 4335 scope.go:122] "RemoveContainer" containerID="ec436a05668041eac1843a8d7d05920e92666717c6ee2eba65e13b7c216d2261" Oct 30 22:15:04.894089 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:15:04.894068 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf_odh-ai-gateway-infra(5ff5482f-138e-4f1b-afad-d4dc9dc99ebf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" podUID="5ff5482f-138e-4f1b-afad-d4dc9dc99ebf" Oct 30 22:15:04.916005 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.915974 4335 scope.go:122] "RemoveContainer" containerID="48ee7d13c3ac00b336cc7b07e2fc9260e7eda6ba3820af16ceea26a58c159b58" Oct 30 22:15:04.931387 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.931367 4335 scope.go:122] "RemoveContainer" containerID="d00a92ea0ea03ad5d92b4aad2d169a032b952c365e3b1c87e87ba6cb4ea199b5" Oct 30 22:15:04.949009 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:04.948986 4335 scope.go:122] "RemoveContainer" containerID="5946574c585819cfb8f2d185b02ee88aaca0c2d6130efa8308018c1029e03c07" Oct 30 22:15:05.060118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:05.060087 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d77bba1-2b42-4033-81af-e7d8a011bba7" path="/var/lib/kubelet/pods/2d77bba1-2b42-4033-81af-e7d8a011bba7/volumes" Oct 30 22:15:05.906340 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:05.906310 4335 scope.go:122] "RemoveContainer" containerID="9858df92f32e1b91e04728286c38f7cc076e2132c59f4d4f9e2b730760af64b7" Oct 30 22:15:05.906805 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:15:05.906662 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc_odh-ai-gateway-infra(0422577b-0c47-4546-80e2-518a849c135b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" podUID="0422577b-0c47-4546-80e2-518a849c135b" Oct 30 22:15:05.908173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:05.908148 4335 scope.go:122] "RemoveContainer" containerID="ec436a05668041eac1843a8d7d05920e92666717c6ee2eba65e13b7c216d2261" Oct 30 22:15:05.908427 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:15:05.908402 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf_odh-ai-gateway-infra(5ff5482f-138e-4f1b-afad-d4dc9dc99ebf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" podUID="5ff5482f-138e-4f1b-afad-d4dc9dc99ebf" Oct 30 22:15:05.909973 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:05.909953 4335 scope.go:122] "RemoveContainer" containerID="2c20cde09f9b9ca3ee7349ef8a4b10beb153bc7ebb9e0c9d5e22f142ac1e056e" Oct 30 22:15:05.910215 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:15:05.910191 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx_odh-ai-gateway-infra(f3a89707-003c-4807-95d3-0bb14607ffdd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" podUID="f3a89707-003c-4807-95d3-0bb14607ffdd" Oct 30 22:15:05.911546 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:05.911529 4335 scope.go:122] "RemoveContainer" containerID="b76908dde95c09dfbf9d42db316e6eb8592e34862bb2efb11f91618887dc7598" Oct 30 22:15:05.911769 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:15:05.911749 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh_odh-ai-gateway-infra(2646e7c5-c250-40b0-af13-d62e8b17aff6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" podUID="2646e7c5-c250-40b0-af13-d62e8b17aff6" Oct 30 22:15:05.913255 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:05.913239 4335 scope.go:122] "RemoveContainer" containerID="7b4bcd32493f0427907fd36330023b2c0560886e73bd76a1374a4c9f87fc3c1d" Oct 30 22:15:05.913468 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:15:05.913451 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq_odh-ai-gateway-infra(f2b5e2a6-85d0-4149-9c26-d65096c5a496)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" podUID="f2b5e2a6-85d0-4149-9c26-d65096c5a496" Oct 30 22:15:06.741577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:06.741549 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22" Oct 30 22:15:15.050839 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:15.050749 4335 scope.go:122] "RemoveContainer" containerID="9858df92f32e1b91e04728286c38f7cc076e2132c59f4d4f9e2b730760af64b7" Oct 30 22:15:15.051414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:15.050860 4335 scope.go:122] "RemoveContainer" containerID="ec436a05668041eac1843a8d7d05920e92666717c6ee2eba65e13b7c216d2261" Oct 30 22:15:15.208138 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:15.208100 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:15:16.050091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:16.050069 4335 scope.go:122] "RemoveContainer" containerID="b76908dde95c09dfbf9d42db316e6eb8592e34862bb2efb11f91618887dc7598" Oct 30 22:15:16.108718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:16.108679 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" event={"ID":"5ff5482f-138e-4f1b-afad-d4dc9dc99ebf","Type":"ContainerStarted","Data":"a7e231a097c5a309878cb4629a860621d535459446894411d2cb5296c4b1f724"} Oct 30 22:15:16.111027 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:16.111003 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" event={"ID":"0422577b-0c47-4546-80e2-518a849c135b","Type":"ContainerStarted","Data":"0bd847722249060c0406d0a39767d236efd97f1f87e5e2dde678677407b7246f"} Oct 30 22:15:17.056201 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:17.056174 4335 scope.go:122] "RemoveContainer" containerID="7b4bcd32493f0427907fd36330023b2c0560886e73bd76a1374a4c9f87fc3c1d" Oct 30 22:15:17.056362 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:17.056302 4335 scope.go:122] "RemoveContainer" containerID="2c20cde09f9b9ca3ee7349ef8a4b10beb153bc7ebb9e0c9d5e22f142ac1e056e" Oct 30 22:15:17.124934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:17.124903 4335 generic.go:356] "Generic (PLEG): container finished" podID="0422577b-0c47-4546-80e2-518a849c135b" containerID="0bd847722249060c0406d0a39767d236efd97f1f87e5e2dde678677407b7246f" exitCode=60 Oct 30 22:15:17.125320 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:17.124993 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" event={"ID":"0422577b-0c47-4546-80e2-518a849c135b","Type":"ContainerDied","Data":"0bd847722249060c0406d0a39767d236efd97f1f87e5e2dde678677407b7246f"} Oct 30 22:15:17.125320 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:17.125072 4335 scope.go:122] "RemoveContainer" containerID="9858df92f32e1b91e04728286c38f7cc076e2132c59f4d4f9e2b730760af64b7" Oct 30 22:15:17.125502 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:17.125481 4335 scope.go:122] "RemoveContainer" containerID="0bd847722249060c0406d0a39767d236efd97f1f87e5e2dde678677407b7246f" Oct 30 22:15:17.125931 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:15:17.125905 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc_odh-ai-gateway-infra(0422577b-0c47-4546-80e2-518a849c135b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" podUID="0422577b-0c47-4546-80e2-518a849c135b" Oct 30 22:15:17.127521 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:17.127502 4335 generic.go:356] "Generic (PLEG): container finished" podID="5ff5482f-138e-4f1b-afad-d4dc9dc99ebf" containerID="a7e231a097c5a309878cb4629a860621d535459446894411d2cb5296c4b1f724" exitCode=60 Oct 30 22:15:17.127609 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:17.127573 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" event={"ID":"5ff5482f-138e-4f1b-afad-d4dc9dc99ebf","Type":"ContainerDied","Data":"a7e231a097c5a309878cb4629a860621d535459446894411d2cb5296c4b1f724"} Oct 30 22:15:17.127962 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:17.127943 4335 scope.go:122] "RemoveContainer" containerID="a7e231a097c5a309878cb4629a860621d535459446894411d2cb5296c4b1f724" Oct 30 22:15:17.128323 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:15:17.128301 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf_odh-ai-gateway-infra(5ff5482f-138e-4f1b-afad-d4dc9dc99ebf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" podUID="5ff5482f-138e-4f1b-afad-d4dc9dc99ebf" Oct 30 22:15:17.129490 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:17.129472 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" event={"ID":"2646e7c5-c250-40b0-af13-d62e8b17aff6","Type":"ContainerStarted","Data":"df8a33fe44fa738cfd0206b6da69d3e33dd59b7fe3a816e71fa5ef5a991dcc7a"} Oct 30 22:15:17.140444 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:17.140417 4335 scope.go:122] "RemoveContainer" containerID="ec436a05668041eac1843a8d7d05920e92666717c6ee2eba65e13b7c216d2261" Oct 30 22:15:17.245996 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:17.245966 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf"] Oct 30 22:15:17.351609 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:17.351585 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc"] Oct 30 22:15:18.143747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.143706 4335 generic.go:356] "Generic (PLEG): container finished" podID="f3a89707-003c-4807-95d3-0bb14607ffdd" containerID="30d87c0740b5b3b5c09e402228206fd2adde2aca2e15ee5a6bea714fb3875cf0" exitCode=60 Oct 30 22:15:18.144183 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.143762 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" event={"ID":"f3a89707-003c-4807-95d3-0bb14607ffdd","Type":"ContainerDied","Data":"30d87c0740b5b3b5c09e402228206fd2adde2aca2e15ee5a6bea714fb3875cf0"} Oct 30 22:15:18.144183 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.143835 4335 scope.go:122] "RemoveContainer" containerID="2c20cde09f9b9ca3ee7349ef8a4b10beb153bc7ebb9e0c9d5e22f142ac1e056e" Oct 30 22:15:18.144928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.144811 4335 scope.go:122] "RemoveContainer" containerID="30d87c0740b5b3b5c09e402228206fd2adde2aca2e15ee5a6bea714fb3875cf0" Oct 30 22:15:18.145727 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:15:18.145267 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx_odh-ai-gateway-infra(f3a89707-003c-4807-95d3-0bb14607ffdd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" podUID="f3a89707-003c-4807-95d3-0bb14607ffdd" Oct 30 22:15:18.147489 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.147464 4335 generic.go:356] "Generic (PLEG): container finished" podID="2646e7c5-c250-40b0-af13-d62e8b17aff6" containerID="df8a33fe44fa738cfd0206b6da69d3e33dd59b7fe3a816e71fa5ef5a991dcc7a" exitCode=60 Oct 30 22:15:18.147627 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.147503 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" event={"ID":"2646e7c5-c250-40b0-af13-d62e8b17aff6","Type":"ContainerDied","Data":"df8a33fe44fa738cfd0206b6da69d3e33dd59b7fe3a816e71fa5ef5a991dcc7a"} Oct 30 22:15:18.148038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.148017 4335 scope.go:122] "RemoveContainer" containerID="df8a33fe44fa738cfd0206b6da69d3e33dd59b7fe3a816e71fa5ef5a991dcc7a" Oct 30 22:15:18.148439 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:15:18.148413 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh_odh-ai-gateway-infra(2646e7c5-c250-40b0-af13-d62e8b17aff6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" podUID="2646e7c5-c250-40b0-af13-d62e8b17aff6" Oct 30 22:15:18.151060 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.151031 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" event={"ID":"f2b5e2a6-85d0-4149-9c26-d65096c5a496","Type":"ContainerStarted","Data":"e45452c7870f40b97373ba202f28cc23aafed971ad53797aa9878f02f773accd"} Oct 30 22:15:18.160239 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.160205 4335 scope.go:122] "RemoveContainer" containerID="b76908dde95c09dfbf9d42db316e6eb8592e34862bb2efb11f91618887dc7598" Oct 30 22:15:18.381227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.381198 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" Oct 30 22:15:18.428055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.428029 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" Oct 30 22:15:18.486486 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.486459 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh"] Oct 30 22:15:18.491247 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.491225 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff5482f-138e-4f1b-afad-d4dc9dc99ebf-kube-api-access-x2p68\" (UniqueName: \"kubernetes.io/projected/5ff5482f-138e-4f1b-afad-d4dc9dc99ebf-kube-api-access-x2p68\") pod \"5ff5482f-138e-4f1b-afad-d4dc9dc99ebf\" (UID: \"5ff5482f-138e-4f1b-afad-d4dc9dc99ebf\") " Oct 30 22:15:18.491370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.491263 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0422577b-0c47-4546-80e2-518a849c135b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0422577b-0c47-4546-80e2-518a849c135b-maas-api-service-ca\") pod \"0422577b-0c47-4546-80e2-518a849c135b\" (UID: \"0422577b-0c47-4546-80e2-518a849c135b\") " Oct 30 22:15:18.491595 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.491556 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ff5482f-138e-4f1b-afad-d4dc9dc99ebf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff5482f-138e-4f1b-afad-d4dc9dc99ebf-maas-api-service-ca\") pod \"5ff5482f-138e-4f1b-afad-d4dc9dc99ebf\" (UID: \"5ff5482f-138e-4f1b-afad-d4dc9dc99ebf\") " Oct 30 22:15:18.491734 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.491706 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0422577b-0c47-4546-80e2-518a849c135b-maas-api-service-ca" pod "0422577b-0c47-4546-80e2-518a849c135b" (UID: "0422577b-0c47-4546-80e2-518a849c135b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:18.491807 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.491768 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0422577b-0c47-4546-80e2-518a849c135b-kube-api-access-rqzl9\" (UniqueName: \"kubernetes.io/projected/0422577b-0c47-4546-80e2-518a849c135b-kube-api-access-rqzl9\") pod \"0422577b-0c47-4546-80e2-518a849c135b\" (UID: \"0422577b-0c47-4546-80e2-518a849c135b\") " Oct 30 22:15:18.491985 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.491944 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff5482f-138e-4f1b-afad-d4dc9dc99ebf-maas-api-service-ca" pod "5ff5482f-138e-4f1b-afad-d4dc9dc99ebf" (UID: "5ff5482f-138e-4f1b-afad-d4dc9dc99ebf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:18.492422 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.492401 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff5482f-138e-4f1b-afad-d4dc9dc99ebf-maas-api-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:18.492507 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.492426 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0422577b-0c47-4546-80e2-518a849c135b-maas-api-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:18.494659 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.494030 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff5482f-138e-4f1b-afad-d4dc9dc99ebf-kube-api-access-x2p68" pod "5ff5482f-138e-4f1b-afad-d4dc9dc99ebf" (UID: "5ff5482f-138e-4f1b-afad-d4dc9dc99ebf"). InnerVolumeSpecName "kube-api-access-x2p68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:18.494659 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.494260 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0422577b-0c47-4546-80e2-518a849c135b-kube-api-access-rqzl9" pod "0422577b-0c47-4546-80e2-518a849c135b" (UID: "0422577b-0c47-4546-80e2-518a849c135b"). InnerVolumeSpecName "kube-api-access-rqzl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:18.594277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.594252 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqzl9\" (UniqueName: \"kubernetes.io/projected/0422577b-0c47-4546-80e2-518a849c135b-kube-api-access-rqzl9\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:18.594277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:18.594278 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2p68\" (UniqueName: \"kubernetes.io/projected/5ff5482f-138e-4f1b-afad-d4dc9dc99ebf-kube-api-access-x2p68\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.169004 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.168976 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" Oct 30 22:15:19.169410 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.169020 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf" event={"ID":"5ff5482f-138e-4f1b-afad-d4dc9dc99ebf","Type":"ContainerDied","Data":"5c95ac5ca9286034c19ef76d9b96908c16c4676b49430a8bef8dd97dc2a4e7c2"} Oct 30 22:15:19.169410 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.169079 4335 scope.go:122] "RemoveContainer" containerID="a7e231a097c5a309878cb4629a860621d535459446894411d2cb5296c4b1f724" Oct 30 22:15:19.175233 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.175204 4335 generic.go:356] "Generic (PLEG): container finished" podID="f2b5e2a6-85d0-4149-9c26-d65096c5a496" containerID="e45452c7870f40b97373ba202f28cc23aafed971ad53797aa9878f02f773accd" exitCode=60 Oct 30 22:15:19.175342 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.175283 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" event={"ID":"f2b5e2a6-85d0-4149-9c26-d65096c5a496","Type":"ContainerDied","Data":"e45452c7870f40b97373ba202f28cc23aafed971ad53797aa9878f02f773accd"} Oct 30 22:15:19.177458 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.175819 4335 scope.go:122] "RemoveContainer" containerID="e45452c7870f40b97373ba202f28cc23aafed971ad53797aa9878f02f773accd" Oct 30 22:15:19.177458 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:15:19.176132 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq_odh-ai-gateway-infra(f2b5e2a6-85d0-4149-9c26-d65096c5a496)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" podUID="f2b5e2a6-85d0-4149-9c26-d65096c5a496" Oct 30 22:15:19.177458 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.177371 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" Oct 30 22:15:19.177458 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.177401 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc" event={"ID":"0422577b-0c47-4546-80e2-518a849c135b","Type":"ContainerDied","Data":"f32a0decd536eae95c7ba6988cde763278f1bc0dc7e458f705f9b2e0f5cb9bdc"} Oct 30 22:15:19.182512 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.182494 4335 scope.go:122] "RemoveContainer" containerID="7b4bcd32493f0427907fd36330023b2c0560886e73bd76a1374a4c9f87fc3c1d" Oct 30 22:15:19.199661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.199637 4335 scope.go:122] "RemoveContainer" containerID="0bd847722249060c0406d0a39767d236efd97f1f87e5e2dde678677407b7246f" Oct 30 22:15:19.274495 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.274467 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx"] Oct 30 22:15:19.412856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.412832 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" Oct 30 22:15:19.456288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.456257 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc"] Oct 30 22:15:19.471767 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.471737 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d7b3c557-29889975-2fbqc"] Oct 30 22:15:19.512174 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.512143 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2646e7c5-c250-40b0-af13-d62e8b17aff6-kube-api-access-ljgrh\" (UniqueName: \"kubernetes.io/projected/2646e7c5-c250-40b0-af13-d62e8b17aff6-kube-api-access-ljgrh\") pod \"2646e7c5-c250-40b0-af13-d62e8b17aff6\" (UID: \"2646e7c5-c250-40b0-af13-d62e8b17aff6\") " Oct 30 22:15:19.512283 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.512213 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2646e7c5-c250-40b0-af13-d62e8b17aff6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2646e7c5-c250-40b0-af13-d62e8b17aff6-maas-api-service-ca\") pod \"2646e7c5-c250-40b0-af13-d62e8b17aff6\" (UID: \"2646e7c5-c250-40b0-af13-d62e8b17aff6\") " Oct 30 22:15:19.512627 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.512589 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2646e7c5-c250-40b0-af13-d62e8b17aff6-maas-api-service-ca" pod "2646e7c5-c250-40b0-af13-d62e8b17aff6" (UID: "2646e7c5-c250-40b0-af13-d62e8b17aff6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:19.512871 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.512814 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2646e7c5-c250-40b0-af13-d62e8b17aff6-maas-api-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.514476 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.514446 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2646e7c5-c250-40b0-af13-d62e8b17aff6-kube-api-access-ljgrh" pod "2646e7c5-c250-40b0-af13-d62e8b17aff6" (UID: "2646e7c5-c250-40b0-af13-d62e8b17aff6"). InnerVolumeSpecName "kube-api-access-ljgrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:19.532842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.532823 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" Oct 30 22:15:19.575320 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.575295 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf"] Oct 30 22:15:19.580446 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.580426 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bf7cbf-29889975-vrvqf"] Oct 30 22:15:19.613526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.613505 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3a89707-003c-4807-95d3-0bb14607ffdd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a89707-003c-4807-95d3-0bb14607ffdd-maas-api-service-ca\") pod \"f3a89707-003c-4807-95d3-0bb14607ffdd\" (UID: \"f3a89707-003c-4807-95d3-0bb14607ffdd\") " Oct 30 22:15:19.613672 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.613576 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3a89707-003c-4807-95d3-0bb14607ffdd-kube-api-access-x7j5n\" (UniqueName: \"kubernetes.io/projected/f3a89707-003c-4807-95d3-0bb14607ffdd-kube-api-access-x7j5n\") pod \"f3a89707-003c-4807-95d3-0bb14607ffdd\" (UID: \"f3a89707-003c-4807-95d3-0bb14607ffdd\") " Oct 30 22:15:19.613945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.613920 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a89707-003c-4807-95d3-0bb14607ffdd-maas-api-service-ca" pod "f3a89707-003c-4807-95d3-0bb14607ffdd" (UID: "f3a89707-003c-4807-95d3-0bb14607ffdd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:19.614244 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.614221 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a89707-003c-4807-95d3-0bb14607ffdd-maas-api-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.614386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.614247 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljgrh\" (UniqueName: \"kubernetes.io/projected/2646e7c5-c250-40b0-af13-d62e8b17aff6-kube-api-access-ljgrh\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.615884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.615859 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a89707-003c-4807-95d3-0bb14607ffdd-kube-api-access-x7j5n" pod "f3a89707-003c-4807-95d3-0bb14607ffdd" (UID: "f3a89707-003c-4807-95d3-0bb14607ffdd"). InnerVolumeSpecName "kube-api-access-x7j5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:19.715181 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.715124 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7j5n\" (UniqueName: \"kubernetes.io/projected/f3a89707-003c-4807-95d3-0bb14607ffdd-kube-api-access-x7j5n\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.739913 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:19.739891 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq"] Oct 30 22:15:20.193358 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:20.193332 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" Oct 30 22:15:20.193861 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:20.193331 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx" event={"ID":"f3a89707-003c-4807-95d3-0bb14607ffdd","Type":"ContainerDied","Data":"582a59109d5385148bb51c468b2015ee8fc5bd2985b3f12348cb3f7958975af6"} Oct 30 22:15:20.193861 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:20.193492 4335 scope.go:122] "RemoveContainer" containerID="30d87c0740b5b3b5c09e402228206fd2adde2aca2e15ee5a6bea714fb3875cf0" Oct 30 22:15:20.195630 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:20.195593 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" Oct 30 22:15:20.195888 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:20.195832 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh" event={"ID":"2646e7c5-c250-40b0-af13-d62e8b17aff6","Type":"ContainerDied","Data":"338a2f5a44324cf812399d5717908bdfe705ef0658881c045c6557caf9b95e2a"} Oct 30 22:15:20.218376 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:20.218351 4335 scope.go:122] "RemoveContainer" containerID="df8a33fe44fa738cfd0206b6da69d3e33dd59b7fe3a816e71fa5ef5a991dcc7a" Oct 30 22:15:20.387640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:20.384668 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx"] Oct 30 22:15:20.401091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:20.394651 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7df73b-29889975-7ljfx"] Oct 30 22:15:20.449361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:20.449310 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" Oct 30 22:15:20.531289 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:20.531267 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b5e2a6-85d0-4149-9c26-d65096c5a496-kube-api-access-kvvm7\" (UniqueName: \"kubernetes.io/projected/f2b5e2a6-85d0-4149-9c26-d65096c5a496-kube-api-access-kvvm7\") pod \"f2b5e2a6-85d0-4149-9c26-d65096c5a496\" (UID: \"f2b5e2a6-85d0-4149-9c26-d65096c5a496\") " Oct 30 22:15:20.531429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:20.531366 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b5e2a6-85d0-4149-9c26-d65096c5a496-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b5e2a6-85d0-4149-9c26-d65096c5a496-maas-api-service-ca\") pod \"f2b5e2a6-85d0-4149-9c26-d65096c5a496\" (UID: \"f2b5e2a6-85d0-4149-9c26-d65096c5a496\") " Oct 30 22:15:20.531782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:20.531754 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b5e2a6-85d0-4149-9c26-d65096c5a496-maas-api-service-ca" pod "f2b5e2a6-85d0-4149-9c26-d65096c5a496" (UID: "f2b5e2a6-85d0-4149-9c26-d65096c5a496"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:20.531927 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:20.531911 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b5e2a6-85d0-4149-9c26-d65096c5a496-maas-api-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:20.533864 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:20.533843 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b5e2a6-85d0-4149-9c26-d65096c5a496-kube-api-access-kvvm7" pod "f2b5e2a6-85d0-4149-9c26-d65096c5a496" (UID: "f2b5e2a6-85d0-4149-9c26-d65096c5a496"). InnerVolumeSpecName "kube-api-access-kvvm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:20.618038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:20.618010 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh"] Oct 30 22:15:20.623141 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:20.623120 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cef074-29889975-cc8wh"] Oct 30 22:15:20.633795 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:20.633766 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvvm7\" (UniqueName: \"kubernetes.io/projected/f2b5e2a6-85d0-4149-9c26-d65096c5a496-kube-api-access-kvvm7\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:21.060185 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:21.060158 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0422577b-0c47-4546-80e2-518a849c135b" path="/var/lib/kubelet/pods/0422577b-0c47-4546-80e2-518a849c135b/volumes" Oct 30 22:15:21.060582 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:21.060568 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2646e7c5-c250-40b0-af13-d62e8b17aff6" path="/var/lib/kubelet/pods/2646e7c5-c250-40b0-af13-d62e8b17aff6/volumes" Oct 30 22:15:21.060992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:21.060978 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff5482f-138e-4f1b-afad-d4dc9dc99ebf" path="/var/lib/kubelet/pods/5ff5482f-138e-4f1b-afad-d4dc9dc99ebf/volumes" Oct 30 22:15:21.061816 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:21.061799 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3a89707-003c-4807-95d3-0bb14607ffdd" path="/var/lib/kubelet/pods/f3a89707-003c-4807-95d3-0bb14607ffdd/volumes" Oct 30 22:15:21.218281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:21.218252 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" Oct 30 22:15:21.218755 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:21.218299 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq" event={"ID":"f2b5e2a6-85d0-4149-9c26-d65096c5a496","Type":"ContainerDied","Data":"f58b2e41ac559be09df91b173b54ba12bb9f446bf33a937ad8f50efc762198e7"} Oct 30 22:15:21.218755 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:21.218354 4335 scope.go:122] "RemoveContainer" containerID="e45452c7870f40b97373ba202f28cc23aafed971ad53797aa9878f02f773accd" Oct 30 22:15:21.391802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:21.391779 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq"] Oct 30 22:15:21.397983 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:21.397962 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7b06b-29889975-t9pmq"] Oct 30 22:15:23.059969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:23.059934 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b5e2a6-85d0-4149-9c26-d65096c5a496" path="/var/lib/kubelet/pods/f2b5e2a6-85d0-4149-9c26-d65096c5a496/volumes" Oct 30 22:15:23.060351 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:23.060333 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh"] Oct 30 22:15:23.102237 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:23.102209 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cbc2e92-ncjqh"] Oct 30 22:15:25.060320 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:25.060290 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc083f7-11a5-476a-8124-5d38fc2b0468" path="/var/lib/kubelet/pods/dcc083f7-11a5-476a-8124-5d38fc2b0468/volumes" Oct 30 22:15:25.322169 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:15:25.322090 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:15:26.053593 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:26.053562 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv"] Oct 30 22:15:26.102466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:26.102436 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5702beb6-gcxfv"] Oct 30 22:15:27.060383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:27.060352 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c0902f1-2ad0-4787-986f-3519210cb168" path="/var/lib/kubelet/pods/7c0902f1-2ad0-4787-986f-3519210cb168/volumes" Oct 30 22:15:28.491520 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:28.491492 4335 scope.go:122] "RemoveContainer" containerID="b7bad8af809eddf970be1e5b3d7444b2bb9ceb1a1f1fa97cb7e784f813ac6607" Oct 30 22:15:28.503044 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:28.503020 4335 scope.go:122] "RemoveContainer" containerID="d1ae9c89b672171b0e0729e1bf52d6758911fa825e5c3396b96cce86608c5780" Oct 30 22:15:28.513089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:28.513071 4335 scope.go:122] "RemoveContainer" containerID="60f57460c318d5f41dd8f26d288eb80dcf9ae1d040f009ff220b040c2a9afa57" Oct 30 22:15:28.523030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:28.523000 4335 scope.go:122] "RemoveContainer" containerID="5b2a9cdd931d3c3332cad1c50b60a62a0a2037b3de9cf103ee2bb06aa3ea59ba" Oct 30 22:15:28.535749 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:28.535725 4335 scope.go:122] "RemoveContainer" containerID="0a9d187d683d679e6aff08a10dfbbe907325d061481068887e3b95ef6c1cb0ad" Oct 30 22:15:28.546922 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:28.546903 4335 scope.go:122] "RemoveContainer" containerID="309c348efdc09ccef6a68b1821b64c94f1adf57713fe72fd70d41177608b68e4" Oct 30 22:15:28.556566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:28.556532 4335 scope.go:122] "RemoveContainer" containerID="7583103ed647c6e278a9d87ee469120a1e61df8992945cfc7c0729451c3402e9" Oct 30 22:15:28.566107 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:28.566083 4335 scope.go:122] "RemoveContainer" containerID="bec126e17e7bbd5dff8a84eb17ccec27b3c659fde28856054b678d82f84fdddb" Oct 30 22:15:28.576014 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:28.575993 4335 scope.go:122] "RemoveContainer" containerID="bf662f801f22f2a918dfe1970495730e346ef6b57ffc5299406ccf3bc5db131d" Oct 30 22:15:28.585370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:28.585350 4335 scope.go:122] "RemoveContainer" containerID="2e2f393fe9398a42ab888425a9bf33b915ab1b45351595390459d31541442476" Oct 30 22:15:28.595029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:28.595014 4335 scope.go:122] "RemoveContainer" containerID="e77d20d818bf9c9dac255d83d804b0c4ae2178e69ad51e68ddf14cf014ced1ba" Oct 30 22:15:28.604426 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:28.604410 4335 scope.go:122] "RemoveContainer" containerID="15c10858297f400351b61f5f9421652a9225c83c27d7371c0ac4ff6fb5c708f8" Oct 30 22:15:31.812851 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.812803 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx"] Oct 30 22:15:31.813804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.813781 4335 cpu_manager.go:414] "containerMap: removing container" podUID="2646e7c5-c250-40b0-af13-d62e8b17aff6" containerName="cleanup" Oct 30 22:15:31.813804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.813804 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2646e7c5-c250-40b0-af13-d62e8b17aff6" containerName="cleanup" Oct 30 22:15:31.813988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.813824 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f2b5e2a6-85d0-4149-9c26-d65096c5a496" containerName="cleanup" Oct 30 22:15:31.813988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.813831 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b5e2a6-85d0-4149-9c26-d65096c5a496" containerName="cleanup" Oct 30 22:15:31.813988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.813843 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f2b5e2a6-85d0-4149-9c26-d65096c5a496" containerName="cleanup" Oct 30 22:15:31.813988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.813851 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b5e2a6-85d0-4149-9c26-d65096c5a496" containerName="cleanup" Oct 30 22:15:31.813988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.813870 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5ff5482f-138e-4f1b-afad-d4dc9dc99ebf" containerName="cleanup" Oct 30 22:15:31.813988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.813877 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff5482f-138e-4f1b-afad-d4dc9dc99ebf" containerName="cleanup" Oct 30 22:15:31.813988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.813888 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5ff5482f-138e-4f1b-afad-d4dc9dc99ebf" containerName="cleanup" Oct 30 22:15:31.813988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.813894 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff5482f-138e-4f1b-afad-d4dc9dc99ebf" containerName="cleanup" Oct 30 22:15:31.813988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.813906 4335 cpu_manager.go:414] "containerMap: removing container" podUID="0422577b-0c47-4546-80e2-518a849c135b" containerName="cleanup" Oct 30 22:15:31.813988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.813913 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0422577b-0c47-4546-80e2-518a849c135b" containerName="cleanup" Oct 30 22:15:31.813988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.813939 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5ff5482f-138e-4f1b-afad-d4dc9dc99ebf" containerName="cleanup" Oct 30 22:15:31.813988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.813947 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff5482f-138e-4f1b-afad-d4dc9dc99ebf" containerName="cleanup" Oct 30 22:15:31.813988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.813973 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f3a89707-003c-4807-95d3-0bb14607ffdd" containerName="cleanup" Oct 30 22:15:31.813988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.813980 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3a89707-003c-4807-95d3-0bb14607ffdd" containerName="cleanup" Oct 30 22:15:31.813988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.813990 4335 cpu_manager.go:414] "containerMap: removing container" podUID="0422577b-0c47-4546-80e2-518a849c135b" containerName="cleanup" Oct 30 22:15:31.814730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.813996 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0422577b-0c47-4546-80e2-518a849c135b" containerName="cleanup" Oct 30 22:15:31.814730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.814011 4335 cpu_manager.go:414] "containerMap: removing container" podUID="2646e7c5-c250-40b0-af13-d62e8b17aff6" containerName="cleanup" Oct 30 22:15:31.814730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.814019 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2646e7c5-c250-40b0-af13-d62e8b17aff6" containerName="cleanup" Oct 30 22:15:31.814730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.814029 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f2b5e2a6-85d0-4149-9c26-d65096c5a496" containerName="cleanup" Oct 30 22:15:31.814730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.814035 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b5e2a6-85d0-4149-9c26-d65096c5a496" containerName="cleanup" Oct 30 22:15:31.814730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.814066 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f3a89707-003c-4807-95d3-0bb14607ffdd" containerName="cleanup" Oct 30 22:15:31.814730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.814074 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3a89707-003c-4807-95d3-0bb14607ffdd" containerName="cleanup" Oct 30 22:15:31.814730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.814255 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff5482f-138e-4f1b-afad-d4dc9dc99ebf" containerName="cleanup" Oct 30 22:15:31.814730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.814269 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3a89707-003c-4807-95d3-0bb14607ffdd" containerName="cleanup" Oct 30 22:15:31.814730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.814277 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="0422577b-0c47-4546-80e2-518a849c135b" containerName="cleanup" Oct 30 22:15:31.814730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.814288 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="2646e7c5-c250-40b0-af13-d62e8b17aff6" containerName="cleanup" Oct 30 22:15:31.814730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.814295 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b5e2a6-85d0-4149-9c26-d65096c5a496" containerName="cleanup" Oct 30 22:15:31.814730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.814308 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3a89707-003c-4807-95d3-0bb14607ffdd" containerName="cleanup" Oct 30 22:15:31.814730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.814319 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="0422577b-0c47-4546-80e2-518a849c135b" containerName="cleanup" Oct 30 22:15:31.814730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.814328 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3a89707-003c-4807-95d3-0bb14607ffdd" containerName="cleanup" Oct 30 22:15:31.814730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.814341 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b5e2a6-85d0-4149-9c26-d65096c5a496" containerName="cleanup" Oct 30 22:15:31.814730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.814351 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="2646e7c5-c250-40b0-af13-d62e8b17aff6" containerName="cleanup" Oct 30 22:15:31.814730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.814360 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff5482f-138e-4f1b-afad-d4dc9dc99ebf" containerName="cleanup" Oct 30 22:15:31.822239 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.822217 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx" Oct 30 22:15:31.874437 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.874399 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx"] Oct 30 22:15:31.948513 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.948451 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:15:31.948714 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.948569 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:15:31.948714 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.948577 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:15:31.948859 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:31.948735 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:15:32.004914 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:32.004868 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx\" (UID: \"b5a25f46-cf7b-4d10-a999-0f42ff5cf432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx" Oct 30 22:15:32.005146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:32.005074 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx\" (UID: \"b5a25f46-cf7b-4d10-a999-0f42ff5cf432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx" Oct 30 22:15:32.005146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:32.005137 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bm6z\" (UniqueName: \"kubernetes.io/projected/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-kube-api-access-5bm6z\") pod \"maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx\" (UID: \"b5a25f46-cf7b-4d10-a999-0f42ff5cf432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx" Oct 30 22:15:32.107453 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:32.107352 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx\" (UID: \"b5a25f46-cf7b-4d10-a999-0f42ff5cf432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx" Oct 30 22:15:32.107649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:32.107484 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx\" (UID: \"b5a25f46-cf7b-4d10-a999-0f42ff5cf432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx" Oct 30 22:15:32.107649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:32.107542 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bm6z\" (UniqueName: \"kubernetes.io/projected/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-kube-api-access-5bm6z\") pod \"maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx\" (UID: \"b5a25f46-cf7b-4d10-a999-0f42ff5cf432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx" Oct 30 22:15:32.108735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:32.108710 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx\" (UID: \"b5a25f46-cf7b-4d10-a999-0f42ff5cf432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx" Oct 30 22:15:32.112140 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:32.110679 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx\" (UID: \"b5a25f46-cf7b-4d10-a999-0f42ff5cf432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx" Oct 30 22:15:32.169218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:32.169187 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bm6z\" (UniqueName: \"kubernetes.io/projected/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-kube-api-access-5bm6z\") pod \"maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx\" (UID: \"b5a25f46-cf7b-4d10-a999-0f42ff5cf432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx" Oct 30 22:15:32.442111 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:32.442023 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx" Oct 30 22:15:32.836968 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:32.836937 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx"] Oct 30 22:15:32.839243 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:15:32.839217 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a25f46_cf7b_4d10_a999_0f42ff5cf432.slice/crio-d4e6781edd1eb2f18be20f3e1118f81b13871ca5ea59a40594e7e12c28115390 WatchSource:0}: Error finding container d4e6781edd1eb2f18be20f3e1118f81b13871ca5ea59a40594e7e12c28115390: Status 404 returned error can't find the container with id d4e6781edd1eb2f18be20f3e1118f81b13871ca5ea59a40594e7e12c28115390 Oct 30 22:15:33.357661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:33.357625 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx" event={"ID":"b5a25f46-cf7b-4d10-a999-0f42ff5cf432","Type":"ContainerStarted","Data":"c3a4d1dd92b26f5084f1be32affa39bb24627ad3d59644f6b7c752de50518127"} Oct 30 22:15:33.357842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:33.357690 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx" event={"ID":"b5a25f46-cf7b-4d10-a999-0f42ff5cf432","Type":"ContainerStarted","Data":"d4e6781edd1eb2f18be20f3e1118f81b13871ca5ea59a40594e7e12c28115390"} Oct 30 22:15:33.474088 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:33.473990 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx" podStartSLOduration=2.4739767759999998 podStartE2EDuration="2.473976776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:33.471571704 +0000 UTC m=+1866.572003462" watchObservedRunningTime="2026-10-30 22:15:33.473976776 +0000 UTC m=+1866.574408507" Oct 30 22:15:34.369433 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:34.369397 4335 generic.go:356] "Generic (PLEG): container finished" podID="b5a25f46-cf7b-4d10-a999-0f42ff5cf432" containerID="c3a4d1dd92b26f5084f1be32affa39bb24627ad3d59644f6b7c752de50518127" exitCode=0 Oct 30 22:15:34.369858 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:34.369450 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx" event={"ID":"b5a25f46-cf7b-4d10-a999-0f42ff5cf432","Type":"ContainerDied","Data":"c3a4d1dd92b26f5084f1be32affa39bb24627ad3d59644f6b7c752de50518127"} Oct 30 22:15:35.059520 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:35.059491 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm"] Oct 30 22:15:35.100863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:35.100828 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7f9b1e-dkplm"] Oct 30 22:15:35.567467 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:35.567441 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx" Oct 30 22:15:35.672996 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:35.672928 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-maas-api-openshift-ca\") pod \"b5a25f46-cf7b-4d10-a999-0f42ff5cf432\" (UID: \"b5a25f46-cf7b-4d10-a999-0f42ff5cf432\") " Oct 30 22:15:35.673187 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:35.673165 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-maas-api-xks-ca\") pod \"b5a25f46-cf7b-4d10-a999-0f42ff5cf432\" (UID: \"b5a25f46-cf7b-4d10-a999-0f42ff5cf432\") " Oct 30 22:15:35.673294 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:35.673277 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-kube-api-access-5bm6z\" (UniqueName: \"kubernetes.io/projected/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-kube-api-access-5bm6z\") pod \"b5a25f46-cf7b-4d10-a999-0f42ff5cf432\" (UID: \"b5a25f46-cf7b-4d10-a999-0f42ff5cf432\") " Oct 30 22:15:35.673340 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:35.673327 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-maas-api-openshift-ca" pod "b5a25f46-cf7b-4d10-a999-0f42ff5cf432" (UID: "b5a25f46-cf7b-4d10-a999-0f42ff5cf432"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:35.673850 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:35.673827 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:35.675418 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:35.675384 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-maas-api-xks-ca" pod "b5a25f46-cf7b-4d10-a999-0f42ff5cf432" (UID: "b5a25f46-cf7b-4d10-a999-0f42ff5cf432"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:35.675537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:35.675449 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-kube-api-access-5bm6z" pod "b5a25f46-cf7b-4d10-a999-0f42ff5cf432" (UID: "b5a25f46-cf7b-4d10-a999-0f42ff5cf432"). InnerVolumeSpecName "kube-api-access-5bm6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:35.775147 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:35.775120 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:35.775147 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:35.775146 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bm6z\" (UniqueName: \"kubernetes.io/projected/b5a25f46-cf7b-4d10-a999-0f42ff5cf432-kube-api-access-5bm6z\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:36.390879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:36.390847 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx" Oct 30 22:15:36.390879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:36.390863 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx" event={"ID":"b5a25f46-cf7b-4d10-a999-0f42ff5cf432","Type":"ContainerDied","Data":"d4e6781edd1eb2f18be20f3e1118f81b13871ca5ea59a40594e7e12c28115390"} Oct 30 22:15:36.391094 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:36.390938 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4e6781edd1eb2f18be20f3e1118f81b13871ca5ea59a40594e7e12c28115390" Oct 30 22:15:37.062859 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:37.062829 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="900d2447-11f1-4eab-95a8-ffca6af2010c" path="/var/lib/kubelet/pods/900d2447-11f1-4eab-95a8-ffca6af2010c/volumes" Oct 30 22:15:38.051069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:38.051035 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf"] Oct 30 22:15:38.101937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:38.101898 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a4871c-lm2nf"] Oct 30 22:15:39.060496 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:39.060464 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80a004d8-e86c-4151-9a8b-eb208f8f0503" path="/var/lib/kubelet/pods/80a004d8-e86c-4151-9a8b-eb208f8f0503/volumes" Oct 30 22:15:40.785647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.783571 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb"] Oct 30 22:15:40.785647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.784864 4335 cpu_manager.go:414] "containerMap: removing container" podUID="0422577b-0c47-4546-80e2-518a849c135b" containerName="cleanup" Oct 30 22:15:40.785647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.784887 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0422577b-0c47-4546-80e2-518a849c135b" containerName="cleanup" Oct 30 22:15:40.785647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.784905 4335 cpu_manager.go:414] "containerMap: removing container" podUID="2646e7c5-c250-40b0-af13-d62e8b17aff6" containerName="cleanup" Oct 30 22:15:40.785647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.784912 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2646e7c5-c250-40b0-af13-d62e8b17aff6" containerName="cleanup" Oct 30 22:15:40.785647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.784934 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f3a89707-003c-4807-95d3-0bb14607ffdd" containerName="cleanup" Oct 30 22:15:40.785647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.784941 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3a89707-003c-4807-95d3-0bb14607ffdd" containerName="cleanup" Oct 30 22:15:40.785647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.785002 4335 cpu_manager.go:414] "containerMap: removing container" podUID="b5a25f46-cf7b-4d10-a999-0f42ff5cf432" containerName="revoke-keys" Oct 30 22:15:40.785647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.785011 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a25f46-cf7b-4d10-a999-0f42ff5cf432" containerName="revoke-keys" Oct 30 22:15:40.785647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.785231 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="0422577b-0c47-4546-80e2-518a849c135b" containerName="cleanup" Oct 30 22:15:40.785647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.785261 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a25f46-cf7b-4d10-a999-0f42ff5cf432" containerName="revoke-keys" Oct 30 22:15:40.785647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.785277 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff5482f-138e-4f1b-afad-d4dc9dc99ebf" containerName="cleanup" Oct 30 22:15:40.785647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.785294 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="2646e7c5-c250-40b0-af13-d62e8b17aff6" containerName="cleanup" Oct 30 22:15:40.785647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.785302 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b5e2a6-85d0-4149-9c26-d65096c5a496" containerName="cleanup" Oct 30 22:15:40.797111 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.797076 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb" Oct 30 22:15:40.875286 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.861958 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb"] Oct 30 22:15:40.940911 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.934937 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-55beb8-dockercfg-8jp8z\"" Oct 30 22:15:40.940911 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.935049 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-55beb8\"" Oct 30 22:15:40.964741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.963904 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833aa422-6f85-413c-ba5e-1befd25b974d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb\" (UID: \"833aa422-6f85-413c-ba5e-1befd25b974d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb" Oct 30 22:15:40.964741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:40.963978 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq4ct\" (UniqueName: \"kubernetes.io/projected/833aa422-6f85-413c-ba5e-1befd25b974d-kube-api-access-mq4ct\") pod \"payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb\" (UID: \"833aa422-6f85-413c-ba5e-1befd25b974d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb" Oct 30 22:15:41.067787 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:41.067742 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833aa422-6f85-413c-ba5e-1befd25b974d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb\" (UID: \"833aa422-6f85-413c-ba5e-1befd25b974d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb" Oct 30 22:15:41.068003 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:41.067811 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq4ct\" (UniqueName: \"kubernetes.io/projected/833aa422-6f85-413c-ba5e-1befd25b974d-kube-api-access-mq4ct\") pod \"payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb\" (UID: \"833aa422-6f85-413c-ba5e-1befd25b974d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb" Oct 30 22:15:41.069279 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:41.069254 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833aa422-6f85-413c-ba5e-1befd25b974d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb\" (UID: \"833aa422-6f85-413c-ba5e-1befd25b974d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb" Oct 30 22:15:41.130300 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:41.130268 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq4ct\" (UniqueName: \"kubernetes.io/projected/833aa422-6f85-413c-ba5e-1befd25b974d-kube-api-access-mq4ct\") pod \"payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb\" (UID: \"833aa422-6f85-413c-ba5e-1befd25b974d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb" Oct 30 22:15:41.429790 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:41.429704 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb" Oct 30 22:15:41.824502 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:41.824471 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb"] Oct 30 22:15:42.470200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:42.470151 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb" event={"ID":"833aa422-6f85-413c-ba5e-1befd25b974d","Type":"ContainerStarted","Data":"5a9bbc83abc1f670d5de592cea2db53b4bd1262751da338b64e4d1d3eff68362"} Oct 30 22:15:42.470401 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:42.470227 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb" Oct 30 22:15:42.470401 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:42.470246 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb" event={"ID":"833aa422-6f85-413c-ba5e-1befd25b974d","Type":"ContainerStarted","Data":"b3e632ea94fb6f909f71f89b00c396f55c068d06f36d3151c0393e3d70a15da1"} Oct 30 22:15:42.643176 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:42.643109 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb" podStartSLOduration=2.64308536 podStartE2EDuration="2.64308536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:42.637009899 +0000 UTC m=+1875.737441631" watchObservedRunningTime="2026-10-30 22:15:42.64308536 +0000 UTC m=+1875.743517092" Oct 30 22:15:52.312268 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:52.312227 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc"] Oct 30 22:15:52.326220 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:52.326186 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc" Oct 30 22:15:52.375213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:52.375184 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc"] Oct 30 22:15:52.430668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:52.417272 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:15:52.430668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:52.417289 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:15:52.430668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:52.417373 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:15:52.430668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:52.428970 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:15:52.467885 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:52.467862 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbpnj\" (UniqueName: \"kubernetes.io/projected/44c4a62d-b9e3-49a2-9f30-02197204f81e-kube-api-access-cbpnj\") pod \"maas-api-revoke-sub-dc9cbb8c-2wmjc\" (UID: \"44c4a62d-b9e3-49a2-9f30-02197204f81e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc" Oct 30 22:15:52.468011 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:52.467907 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c4a62d-b9e3-49a2-9f30-02197204f81e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc9cbb8c-2wmjc\" (UID: \"44c4a62d-b9e3-49a2-9f30-02197204f81e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc" Oct 30 22:15:52.468113 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:52.468095 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c4a62d-b9e3-49a2-9f30-02197204f81e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc9cbb8c-2wmjc\" (UID: \"44c4a62d-b9e3-49a2-9f30-02197204f81e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc" Oct 30 22:15:52.569723 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:52.569647 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c4a62d-b9e3-49a2-9f30-02197204f81e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc9cbb8c-2wmjc\" (UID: \"44c4a62d-b9e3-49a2-9f30-02197204f81e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc" Oct 30 22:15:52.569869 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:52.569824 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c4a62d-b9e3-49a2-9f30-02197204f81e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc9cbb8c-2wmjc\" (UID: \"44c4a62d-b9e3-49a2-9f30-02197204f81e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc" Oct 30 22:15:52.570020 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:52.570004 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbpnj\" (UniqueName: \"kubernetes.io/projected/44c4a62d-b9e3-49a2-9f30-02197204f81e-kube-api-access-cbpnj\") pod \"maas-api-revoke-sub-dc9cbb8c-2wmjc\" (UID: \"44c4a62d-b9e3-49a2-9f30-02197204f81e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc" Oct 30 22:15:52.570249 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:52.570231 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c4a62d-b9e3-49a2-9f30-02197204f81e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc9cbb8c-2wmjc\" (UID: \"44c4a62d-b9e3-49a2-9f30-02197204f81e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc" Oct 30 22:15:52.572253 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:52.572226 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c4a62d-b9e3-49a2-9f30-02197204f81e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc9cbb8c-2wmjc\" (UID: \"44c4a62d-b9e3-49a2-9f30-02197204f81e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc" Oct 30 22:15:52.644846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:52.644814 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbpnj\" (UniqueName: \"kubernetes.io/projected/44c4a62d-b9e3-49a2-9f30-02197204f81e-kube-api-access-cbpnj\") pod \"maas-api-revoke-sub-dc9cbb8c-2wmjc\" (UID: \"44c4a62d-b9e3-49a2-9f30-02197204f81e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc" Oct 30 22:15:52.942392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:52.942298 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc" Oct 30 22:15:53.340863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:53.340001 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc"] Oct 30 22:15:53.341374 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:15:53.341278 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c4a62d_b9e3_49a2_9f30_02197204f81e.slice/crio-d09ae5c81e579f7da805eccf6c6cd1f5f0957143592126bb557990a095959668 WatchSource:0}: Error finding container d09ae5c81e579f7da805eccf6c6cd1f5f0957143592126bb557990a095959668: Status 404 returned error can't find the container with id d09ae5c81e579f7da805eccf6c6cd1f5f0957143592126bb557990a095959668 Oct 30 22:15:53.483262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:53.483235 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb" Oct 30 22:15:53.637508 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:53.637470 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc" event={"ID":"44c4a62d-b9e3-49a2-9f30-02197204f81e","Type":"ContainerStarted","Data":"468cb04fce24a1e9bec91d55a23923ec30a8469b0e2830d760da01fd16f805e4"} Oct 30 22:15:53.637692 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:53.637535 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc" event={"ID":"44c4a62d-b9e3-49a2-9f30-02197204f81e","Type":"ContainerStarted","Data":"d09ae5c81e579f7da805eccf6c6cd1f5f0957143592126bb557990a095959668"} Oct 30 22:15:53.754914 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:53.754857 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc" podStartSLOduration=1.754843202 podStartE2EDuration="1.754843202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:53.753355164 +0000 UTC m=+1886.853786898" watchObservedRunningTime="2026-10-30 22:15:53.754843202 +0000 UTC m=+1886.855274934" Oct 30 22:15:54.649091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:54.649012 4335 generic.go:356] "Generic (PLEG): container finished" podID="44c4a62d-b9e3-49a2-9f30-02197204f81e" containerID="468cb04fce24a1e9bec91d55a23923ec30a8469b0e2830d760da01fd16f805e4" exitCode=0 Oct 30 22:15:54.649483 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:54.649102 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc" event={"ID":"44c4a62d-b9e3-49a2-9f30-02197204f81e","Type":"ContainerDied","Data":"468cb04fce24a1e9bec91d55a23923ec30a8469b0e2830d760da01fd16f805e4"} Oct 30 22:15:55.322744 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:15:55.322701 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:15:55.846453 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:55.846427 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc" Oct 30 22:15:56.026127 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:56.026035 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c4a62d-b9e3-49a2-9f30-02197204f81e-kube-api-access-cbpnj\" (UniqueName: \"kubernetes.io/projected/44c4a62d-b9e3-49a2-9f30-02197204f81e-kube-api-access-cbpnj\") pod \"44c4a62d-b9e3-49a2-9f30-02197204f81e\" (UID: \"44c4a62d-b9e3-49a2-9f30-02197204f81e\") " Oct 30 22:15:56.026297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:56.026226 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44c4a62d-b9e3-49a2-9f30-02197204f81e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c4a62d-b9e3-49a2-9f30-02197204f81e-maas-api-openshift-ca\") pod \"44c4a62d-b9e3-49a2-9f30-02197204f81e\" (UID: \"44c4a62d-b9e3-49a2-9f30-02197204f81e\") " Oct 30 22:15:56.026401 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:56.026383 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44c4a62d-b9e3-49a2-9f30-02197204f81e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c4a62d-b9e3-49a2-9f30-02197204f81e-maas-api-xks-ca\") pod \"44c4a62d-b9e3-49a2-9f30-02197204f81e\" (UID: \"44c4a62d-b9e3-49a2-9f30-02197204f81e\") " Oct 30 22:15:56.026772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:56.026745 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c4a62d-b9e3-49a2-9f30-02197204f81e-maas-api-openshift-ca" pod "44c4a62d-b9e3-49a2-9f30-02197204f81e" (UID: "44c4a62d-b9e3-49a2-9f30-02197204f81e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:56.027456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:56.027434 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c4a62d-b9e3-49a2-9f30-02197204f81e-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:56.029442 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:56.029375 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c4a62d-b9e3-49a2-9f30-02197204f81e-maas-api-xks-ca" pod "44c4a62d-b9e3-49a2-9f30-02197204f81e" (UID: "44c4a62d-b9e3-49a2-9f30-02197204f81e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:56.030806 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:56.030778 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c4a62d-b9e3-49a2-9f30-02197204f81e-kube-api-access-cbpnj" pod "44c4a62d-b9e3-49a2-9f30-02197204f81e" (UID: "44c4a62d-b9e3-49a2-9f30-02197204f81e"). InnerVolumeSpecName "kube-api-access-cbpnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:56.129249 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:56.129213 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbpnj\" (UniqueName: \"kubernetes.io/projected/44c4a62d-b9e3-49a2-9f30-02197204f81e-kube-api-access-cbpnj\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:56.129249 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:56.129241 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c4a62d-b9e3-49a2-9f30-02197204f81e-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:56.676412 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:56.676381 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc" Oct 30 22:15:56.676412 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:56.676387 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc" event={"ID":"44c4a62d-b9e3-49a2-9f30-02197204f81e","Type":"ContainerDied","Data":"d09ae5c81e579f7da805eccf6c6cd1f5f0957143592126bb557990a095959668"} Oct 30 22:15:56.676666 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:15:56.676467 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d09ae5c81e579f7da805eccf6c6cd1f5f0957143592126bb557990a095959668" Oct 30 22:16:02.052924 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:02.052890 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x"] Oct 30 22:16:02.101794 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:02.101765 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9cc2135-q9l2x"] Oct 30 22:16:03.060546 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:03.060514 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cbf29c0-be44-4c3b-a196-b1af8a4f27b7" path="/var/lib/kubelet/pods/4cbf29c0-be44-4c3b-a196-b1af8a4f27b7/volumes" Oct 30 22:16:06.926702 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:06.926667 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx"] Oct 30 22:16:06.975538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:06.975508 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7b3c557-qdbdx"] Oct 30 22:16:07.070654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:07.070603 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a25f46-cf7b-4d10-a999-0f42ff5cf432" path="/var/lib/kubelet/pods/b5a25f46-cf7b-4d10-a999-0f42ff5cf432/volumes" Oct 30 22:16:10.071704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:10.071672 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d"] Oct 30 22:16:10.120629 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:10.120595 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22e8244-2qh4d"] Oct 30 22:16:11.064591 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:11.064554 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0955b107-e6d3-495d-9ade-67a6f9ada6bb" path="/var/lib/kubelet/pods/0955b107-e6d3-495d-9ade-67a6f9ada6bb/volumes" Oct 30 22:16:12.062258 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:12.062213 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4"] Oct 30 22:16:12.111711 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:12.111685 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc3318a-tmvb4"] Oct 30 22:16:13.061729 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:13.061698 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c05860ee-b236-4dfa-bcb1-a38795a2c140" path="/var/lib/kubelet/pods/c05860ee-b236-4dfa-bcb1-a38795a2c140/volumes" Oct 30 22:16:13.161027 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:13.160996 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw"] Oct 30 22:16:13.161498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:13.161191 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" podUID="6c79ac4e-ddf1-41d0-a8ac-326974828e18" containerName="istio-proxy" containerID="cri-o://541c4f8f00dff5301f42a0547ccf95b560d883f3bf5cf167d72329ffe244d9a6" gracePeriod=30 Oct 30 22:16:13.208795 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:13.208766 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25"] Oct 30 22:16:13.208964 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:13.208945 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerName="istio-proxy" containerID="cri-o://de8cd9446ff513c95801d372d3e1dab934ecfcc1ef57b33bccbd20e8bb29018f" gracePeriod=30 Oct 30 22:16:13.214007 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:13.213981 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx"] Oct 30 22:16:13.214178 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:13.214147 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerName="istio-proxy" containerID="cri-o://6b183b991dbf27a40f5a78cf78439db5350f3289c03775689637272f8fb9617b" gracePeriod=30 Oct 30 22:16:13.258110 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:13.258084 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78"] Oct 30 22:16:13.258315 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:13.258290 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" podUID="d6978eea-f24f-4d30-83b0-85b0015baf53" containerName="istio-proxy" containerID="cri-o://3d4598e61d58fabf8822a679c60dcfa7e9e7dd21f36f77a7acfaee709fc73041" gracePeriod=30 Oct 30 22:16:18.603567 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.603164 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:16:18.643350 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.640013 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:16:18.701148 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.701098 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-data\") pod \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " Oct 30 22:16:18.701326 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.701211 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c79ac4e-ddf1-41d0-a8ac-326974828e18-kube-api-access-85rf7\" (UniqueName: \"kubernetes.io/projected/6c79ac4e-ddf1-41d0-a8ac-326974828e18-kube-api-access-85rf7\") pod \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " Oct 30 22:16:18.701326 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.701252 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-token\" (UniqueName: \"kubernetes.io/projected/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-token\") pod \"d6978eea-f24f-4d30-83b0-85b0015baf53\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " Oct 30 22:16:18.701447 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.701369 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-podinfo\") pod \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " Oct 30 22:16:18.701447 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.701434 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6978eea-f24f-4d30-83b0-85b0015baf53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6978eea-f24f-4d30-83b0-85b0015baf53-istiod-ca-cert\") pod \"d6978eea-f24f-4d30-83b0-85b0015baf53\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " Oct 30 22:16:18.701554 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.701466 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istiod-ca-cert\") pod \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " Oct 30 22:16:18.701554 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.701501 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-envoy\") pod \"d6978eea-f24f-4d30-83b0-85b0015baf53\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " Oct 30 22:16:18.701689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.701573 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-workload-certs\") pod \"d6978eea-f24f-4d30-83b0-85b0015baf53\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " Oct 30 22:16:18.701689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.701666 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-data\") pod \"d6978eea-f24f-4d30-83b0-85b0015baf53\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " Oct 30 22:16:18.701796 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.701710 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-envoy\") pod \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " Oct 30 22:16:18.701796 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.701748 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6978eea-f24f-4d30-83b0-85b0015baf53-kube-api-access-d5ldd\" (UniqueName: \"kubernetes.io/projected/d6978eea-f24f-4d30-83b0-85b0015baf53-kube-api-access-d5ldd\") pod \"d6978eea-f24f-4d30-83b0-85b0015baf53\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " Oct 30 22:16:18.701964 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.701835 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-workload-certs\") pod \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " Oct 30 22:16:18.701964 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.701872 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-workload-socket\") pod \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " Oct 30 22:16:18.701964 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.701918 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-workload-socket\") pod \"d6978eea-f24f-4d30-83b0-85b0015baf53\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " Oct 30 22:16:18.702118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.702005 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-token\" (UniqueName: \"kubernetes.io/projected/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-token\") pod \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " Oct 30 22:16:18.702118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.702095 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-podinfo\") pod \"d6978eea-f24f-4d30-83b0-85b0015baf53\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " Oct 30 22:16:18.702225 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.702197 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-credential-socket\") pod \"d6978eea-f24f-4d30-83b0-85b0015baf53\" (UID: \"d6978eea-f24f-4d30-83b0-85b0015baf53\") " Oct 30 22:16:18.702281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.702239 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-credential-socket\") pod \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\" (UID: \"6c79ac4e-ddf1-41d0-a8ac-326974828e18\") " Oct 30 22:16:18.705385 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.705288 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-credential-socket" pod "6c79ac4e-ddf1-41d0-a8ac-326974828e18" (UID: "6c79ac4e-ddf1-41d0-a8ac-326974828e18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.705636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.705548 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-data" pod "d6978eea-f24f-4d30-83b0-85b0015baf53" (UID: "d6978eea-f24f-4d30-83b0-85b0015baf53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.706115 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.705989 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-data" pod "6c79ac4e-ddf1-41d0-a8ac-326974828e18" (UID: "6c79ac4e-ddf1-41d0-a8ac-326974828e18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.706209 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.706146 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-workload-socket" pod "d6978eea-f24f-4d30-83b0-85b0015baf53" (UID: "d6978eea-f24f-4d30-83b0-85b0015baf53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.707265 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.707239 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-workload-certs" pod "6c79ac4e-ddf1-41d0-a8ac-326974828e18" (UID: "6c79ac4e-ddf1-41d0-a8ac-326974828e18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.707558 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.707535 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6978eea-f24f-4d30-83b0-85b0015baf53-istiod-ca-cert" pod "d6978eea-f24f-4d30-83b0-85b0015baf53" (UID: "d6978eea-f24f-4d30-83b0-85b0015baf53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:18.709317 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.709293 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-workload-socket" pod "6c79ac4e-ddf1-41d0-a8ac-326974828e18" (UID: "6c79ac4e-ddf1-41d0-a8ac-326974828e18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.713595 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.713568 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-podinfo" pod "d6978eea-f24f-4d30-83b0-85b0015baf53" (UID: "d6978eea-f24f-4d30-83b0-85b0015baf53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:18.715926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.715883 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-token" pod "6c79ac4e-ddf1-41d0-a8ac-326974828e18" (UID: "6c79ac4e-ddf1-41d0-a8ac-326974828e18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:18.716100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.716080 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-credential-socket" pod "d6978eea-f24f-4d30-83b0-85b0015baf53" (UID: "d6978eea-f24f-4d30-83b0-85b0015baf53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.717265 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.717053 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6978eea-f24f-4d30-83b0-85b0015baf53-kube-api-access-d5ldd" pod "d6978eea-f24f-4d30-83b0-85b0015baf53" (UID: "d6978eea-f24f-4d30-83b0-85b0015baf53"). InnerVolumeSpecName "kube-api-access-d5ldd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:18.717743 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.717493 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istiod-ca-cert" pod "6c79ac4e-ddf1-41d0-a8ac-326974828e18" (UID: "6c79ac4e-ddf1-41d0-a8ac-326974828e18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:18.718111 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.717884 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-workload-certs" pod "d6978eea-f24f-4d30-83b0-85b0015baf53" (UID: "d6978eea-f24f-4d30-83b0-85b0015baf53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.720662 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.719106 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c79ac4e-ddf1-41d0-a8ac-326974828e18-kube-api-access-85rf7" pod "6c79ac4e-ddf1-41d0-a8ac-326974828e18" (UID: "6c79ac4e-ddf1-41d0-a8ac-326974828e18"). InnerVolumeSpecName "kube-api-access-85rf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:18.728959 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.728933 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-token" pod "d6978eea-f24f-4d30-83b0-85b0015baf53" (UID: "d6978eea-f24f-4d30-83b0-85b0015baf53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:18.734641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.733907 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-podinfo" pod "6c79ac4e-ddf1-41d0-a8ac-326974828e18" (UID: "6c79ac4e-ddf1-41d0-a8ac-326974828e18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:18.735440 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.735413 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-envoy" pod "d6978eea-f24f-4d30-83b0-85b0015baf53" (UID: "d6978eea-f24f-4d30-83b0-85b0015baf53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.735440 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.735428 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-envoy" pod "6c79ac4e-ddf1-41d0-a8ac-326974828e18" (UID: "6c79ac4e-ddf1-41d0-a8ac-326974828e18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.746605 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.746584 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:16:18.749560 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.749542 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:16:18.806312 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.806288 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-data\") pod \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " Oct 30 22:16:18.806443 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.806339 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-token\" (UniqueName: \"kubernetes.io/projected/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-token\") pod \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " Oct 30 22:16:18.806443 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.806371 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-data\") pod \"c55f64be-7022-43a7-a419-cbf1ab06223a\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " Oct 30 22:16:18.806443 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.806418 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-workload-certs\") pod \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " Oct 30 22:16:18.815480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.806469 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-kube-api-access-llr6v\" (UniqueName: \"kubernetes.io/projected/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-kube-api-access-llr6v\") pod \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " Oct 30 22:16:18.815480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.806542 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-envoy\") pod \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " Oct 30 22:16:18.815480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.806573 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-credential-socket\") pod \"c55f64be-7022-43a7-a419-cbf1ab06223a\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " Oct 30 22:16:18.815480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.806629 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c55f64be-7022-43a7-a419-cbf1ab06223a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55f64be-7022-43a7-a419-cbf1ab06223a-istiod-ca-cert\") pod \"c55f64be-7022-43a7-a419-cbf1ab06223a\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " Oct 30 22:16:18.815480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.806666 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istiod-ca-cert\") pod \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " Oct 30 22:16:18.815480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.806701 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-token\" (UniqueName: \"kubernetes.io/projected/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-token\") pod \"c55f64be-7022-43a7-a419-cbf1ab06223a\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " Oct 30 22:16:18.815480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.806737 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-credential-socket\") pod \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " Oct 30 22:16:18.815480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.806767 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-envoy\") pod \"c55f64be-7022-43a7-a419-cbf1ab06223a\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " Oct 30 22:16:18.815480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.806796 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-workload-certs\") pod \"c55f64be-7022-43a7-a419-cbf1ab06223a\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " Oct 30 22:16:18.815480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.806970 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-podinfo\") pod \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " Oct 30 22:16:18.815480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.807008 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-workload-socket\") pod \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\" (UID: \"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1\") " Oct 30 22:16:18.815480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.807064 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-podinfo\") pod \"c55f64be-7022-43a7-a419-cbf1ab06223a\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " Oct 30 22:16:18.815480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.807181 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-workload-socket\") pod \"c55f64be-7022-43a7-a419-cbf1ab06223a\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " Oct 30 22:16:18.815480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.807235 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c55f64be-7022-43a7-a419-cbf1ab06223a-kube-api-access-gfw99\" (UniqueName: \"kubernetes.io/projected/c55f64be-7022-43a7-a419-cbf1ab06223a-kube-api-access-gfw99\") pod \"c55f64be-7022-43a7-a419-cbf1ab06223a\" (UID: \"c55f64be-7022-43a7-a419-cbf1ab06223a\") " Oct 30 22:16:18.815480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.808073 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-workload-certs" pod "c55f64be-7022-43a7-a419-cbf1ab06223a" (UID: "c55f64be-7022-43a7-a419-cbf1ab06223a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.815480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.808108 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istiod-ca-cert" pod "e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" (UID: "e372cc9b-a3ca-4ad2-975d-cdaa392a87d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.808862 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-data" pod "e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" (UID: "e372cc9b-a3ca-4ad2-975d-cdaa392a87d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.809215 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-credential-socket" pod "e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" (UID: "e372cc9b-a3ca-4ad2-975d-cdaa392a87d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.810072 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-workload-socket" pod "e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" (UID: "e372cc9b-a3ca-4ad2-975d-cdaa392a87d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.810657 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-data" pod "c55f64be-7022-43a7-a419-cbf1ab06223a" (UID: "c55f64be-7022-43a7-a419-cbf1ab06223a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.810871 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-workload-certs" pod "e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" (UID: "e372cc9b-a3ca-4ad2-975d-cdaa392a87d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.811119 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-credential-socket" pod "c55f64be-7022-43a7-a419-cbf1ab06223a" (UID: "c55f64be-7022-43a7-a419-cbf1ab06223a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.811337 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-workload-socket" pod "c55f64be-7022-43a7-a419-cbf1ab06223a" (UID: "c55f64be-7022-43a7-a419-cbf1ab06223a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.811804 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-token" pod "c55f64be-7022-43a7-a419-cbf1ab06223a" (UID: "c55f64be-7022-43a7-a419-cbf1ab06223a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.812428 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-token" pod "e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" (UID: "e372cc9b-a3ca-4ad2-975d-cdaa392a87d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.814485 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55f64be-7022-43a7-a419-cbf1ab06223a-istiod-ca-cert" pod "c55f64be-7022-43a7-a419-cbf1ab06223a" (UID: "c55f64be-7022-43a7-a419-cbf1ab06223a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.815994 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816011 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816020 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816032 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5ldd\" (UniqueName: \"kubernetes.io/projected/d6978eea-f24f-4d30-83b0-85b0015baf53-kube-api-access-d5ldd\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816041 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816050 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816058 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816067 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816074 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816082 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816090 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816098 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816105 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816115 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816123 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816131 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55f64be-7022-43a7-a419-cbf1ab06223a-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816139 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816146 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816154 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816163 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816177 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816194 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816202 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816211 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85rf7\" (UniqueName: \"kubernetes.io/projected/6c79ac4e-ddf1-41d0-a8ac-326974828e18-kube-api-access-85rf7\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816218 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816227 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816235 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816244 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6978eea-f24f-4d30-83b0-85b0015baf53-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816253 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c79ac4e-ddf1-41d0-a8ac-326974828e18-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.816384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816261 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6978eea-f24f-4d30-83b0-85b0015baf53-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.818333 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.816504 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-podinfo" pod "c55f64be-7022-43a7-a419-cbf1ab06223a" (UID: "c55f64be-7022-43a7-a419-cbf1ab06223a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:18.818333 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.817052 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-envoy" pod "e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" (UID: "e372cc9b-a3ca-4ad2-975d-cdaa392a87d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.818333 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.817051 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-podinfo" pod "e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" (UID: "e372cc9b-a3ca-4ad2-975d-cdaa392a87d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:18.818333 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.817188 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-envoy" pod "c55f64be-7022-43a7-a419-cbf1ab06223a" (UID: "c55f64be-7022-43a7-a419-cbf1ab06223a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.818333 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.817192 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-kube-api-access-llr6v" pod "e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" (UID: "e372cc9b-a3ca-4ad2-975d-cdaa392a87d1"). InnerVolumeSpecName "kube-api-access-llr6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:18.819291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.819265 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55f64be-7022-43a7-a419-cbf1ab06223a-kube-api-access-gfw99" pod "c55f64be-7022-43a7-a419-cbf1ab06223a" (UID: "c55f64be-7022-43a7-a419-cbf1ab06223a"). InnerVolumeSpecName "kube-api-access-gfw99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:18.918091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.918042 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.918091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.918073 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.918091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.918090 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfw99\" (UniqueName: \"kubernetes.io/projected/c55f64be-7022-43a7-a419-cbf1ab06223a-kube-api-access-gfw99\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.918533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.918146 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llr6v\" (UniqueName: \"kubernetes.io/projected/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-kube-api-access-llr6v\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.918533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.918163 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.918533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.918176 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55f64be-7022-43a7-a419-cbf1ab06223a-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.925352 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.925315 4335 generic.go:356] "Generic (PLEG): container finished" podID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerID="de8cd9446ff513c95801d372d3e1dab934ecfcc1ef57b33bccbd20e8bb29018f" exitCode=0 Oct 30 22:16:18.925482 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.925460 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" event={"ID":"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1","Type":"ContainerDied","Data":"de8cd9446ff513c95801d372d3e1dab934ecfcc1ef57b33bccbd20e8bb29018f"} Oct 30 22:16:18.925564 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.925550 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" event={"ID":"e372cc9b-a3ca-4ad2-975d-cdaa392a87d1","Type":"ContainerDied","Data":"07798d4fd0c4fa852b723b8fcf0c6f0cfa043f5e310c7c0b8307c12a5c4757f5"} Oct 30 22:16:18.925641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.925584 4335 scope.go:122] "RemoveContainer" containerID="de8cd9446ff513c95801d372d3e1dab934ecfcc1ef57b33bccbd20e8bb29018f" Oct 30 22:16:18.925825 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.925807 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25" Oct 30 22:16:18.945845 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.938468 4335 generic.go:356] "Generic (PLEG): container finished" podID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerID="6b183b991dbf27a40f5a78cf78439db5350f3289c03775689637272f8fb9617b" exitCode=0 Oct 30 22:16:18.945845 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.938552 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" event={"ID":"c55f64be-7022-43a7-a419-cbf1ab06223a","Type":"ContainerDied","Data":"6b183b991dbf27a40f5a78cf78439db5350f3289c03775689637272f8fb9617b"} Oct 30 22:16:18.945845 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.938594 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" event={"ID":"c55f64be-7022-43a7-a419-cbf1ab06223a","Type":"ContainerDied","Data":"e70d56d6f8819fd4318a0bccee1af826b07fe450f30fc319aeb64fa71f05950d"} Oct 30 22:16:18.945845 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.938727 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx" Oct 30 22:16:18.945845 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.944508 4335 generic.go:356] "Generic (PLEG): container finished" podID="d6978eea-f24f-4d30-83b0-85b0015baf53" containerID="3d4598e61d58fabf8822a679c60dcfa7e9e7dd21f36f77a7acfaee709fc73041" exitCode=0 Oct 30 22:16:18.945845 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.944677 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" Oct 30 22:16:18.949318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.948306 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" event={"ID":"d6978eea-f24f-4d30-83b0-85b0015baf53","Type":"ContainerDied","Data":"3d4598e61d58fabf8822a679c60dcfa7e9e7dd21f36f77a7acfaee709fc73041"} Oct 30 22:16:18.949318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.948350 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78" event={"ID":"d6978eea-f24f-4d30-83b0-85b0015baf53","Type":"ContainerDied","Data":"4e14eb3d811f5d9f97738c0dcebfa5983dab317570d00ce5e383c3de9db89fa1"} Oct 30 22:16:18.954319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.954297 4335 generic.go:356] "Generic (PLEG): container finished" podID="6c79ac4e-ddf1-41d0-a8ac-326974828e18" containerID="541c4f8f00dff5301f42a0547ccf95b560d883f3bf5cf167d72329ffe244d9a6" exitCode=0 Oct 30 22:16:18.954513 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.954498 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" Oct 30 22:16:18.954889 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.954566 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" event={"ID":"6c79ac4e-ddf1-41d0-a8ac-326974828e18","Type":"ContainerDied","Data":"541c4f8f00dff5301f42a0547ccf95b560d883f3bf5cf167d72329ffe244d9a6"} Oct 30 22:16:18.954980 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.954918 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw" event={"ID":"6c79ac4e-ddf1-41d0-a8ac-326974828e18","Type":"ContainerDied","Data":"c7df01b7402c7f79412d263c8f66674d9707d555f18e1972fd224d50be9e13dc"} Oct 30 22:16:18.956057 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.956029 4335 scope.go:122] "RemoveContainer" containerID="de8cd9446ff513c95801d372d3e1dab934ecfcc1ef57b33bccbd20e8bb29018f" Oct 30 22:16:18.956326 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:16:18.956306 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8cd9446ff513c95801d372d3e1dab934ecfcc1ef57b33bccbd20e8bb29018f\": container with ID starting with de8cd9446ff513c95801d372d3e1dab934ecfcc1ef57b33bccbd20e8bb29018f not found: ID does not exist" containerID="de8cd9446ff513c95801d372d3e1dab934ecfcc1ef57b33bccbd20e8bb29018f" Oct 30 22:16:18.956421 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.956338 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8cd9446ff513c95801d372d3e1dab934ecfcc1ef57b33bccbd20e8bb29018f"} err="failed to get container status \"de8cd9446ff513c95801d372d3e1dab934ecfcc1ef57b33bccbd20e8bb29018f\": rpc error: code = NotFound desc = could not find container \"de8cd9446ff513c95801d372d3e1dab934ecfcc1ef57b33bccbd20e8bb29018f\": container with ID starting with de8cd9446ff513c95801d372d3e1dab934ecfcc1ef57b33bccbd20e8bb29018f not found: ID does not exist" Oct 30 22:16:18.956421 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.956358 4335 scope.go:122] "RemoveContainer" containerID="6b183b991dbf27a40f5a78cf78439db5350f3289c03775689637272f8fb9617b" Oct 30 22:16:18.970203 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.970178 4335 scope.go:122] "RemoveContainer" containerID="6b183b991dbf27a40f5a78cf78439db5350f3289c03775689637272f8fb9617b" Oct 30 22:16:18.971012 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:16:18.970989 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b183b991dbf27a40f5a78cf78439db5350f3289c03775689637272f8fb9617b\": container with ID starting with 6b183b991dbf27a40f5a78cf78439db5350f3289c03775689637272f8fb9617b not found: ID does not exist" containerID="6b183b991dbf27a40f5a78cf78439db5350f3289c03775689637272f8fb9617b" Oct 30 22:16:18.971137 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.971016 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b183b991dbf27a40f5a78cf78439db5350f3289c03775689637272f8fb9617b"} err="failed to get container status \"6b183b991dbf27a40f5a78cf78439db5350f3289c03775689637272f8fb9617b\": rpc error: code = NotFound desc = could not find container \"6b183b991dbf27a40f5a78cf78439db5350f3289c03775689637272f8fb9617b\": container with ID starting with 6b183b991dbf27a40f5a78cf78439db5350f3289c03775689637272f8fb9617b not found: ID does not exist" Oct 30 22:16:18.971137 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.971031 4335 scope.go:122] "RemoveContainer" containerID="3d4598e61d58fabf8822a679c60dcfa7e9e7dd21f36f77a7acfaee709fc73041" Oct 30 22:16:18.987197 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.987175 4335 scope.go:122] "RemoveContainer" containerID="3d4598e61d58fabf8822a679c60dcfa7e9e7dd21f36f77a7acfaee709fc73041" Oct 30 22:16:18.987464 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:16:18.987444 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4598e61d58fabf8822a679c60dcfa7e9e7dd21f36f77a7acfaee709fc73041\": container with ID starting with 3d4598e61d58fabf8822a679c60dcfa7e9e7dd21f36f77a7acfaee709fc73041 not found: ID does not exist" containerID="3d4598e61d58fabf8822a679c60dcfa7e9e7dd21f36f77a7acfaee709fc73041" Oct 30 22:16:18.987537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.987473 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4598e61d58fabf8822a679c60dcfa7e9e7dd21f36f77a7acfaee709fc73041"} err="failed to get container status \"3d4598e61d58fabf8822a679c60dcfa7e9e7dd21f36f77a7acfaee709fc73041\": rpc error: code = NotFound desc = could not find container \"3d4598e61d58fabf8822a679c60dcfa7e9e7dd21f36f77a7acfaee709fc73041\": container with ID starting with 3d4598e61d58fabf8822a679c60dcfa7e9e7dd21f36f77a7acfaee709fc73041 not found: ID does not exist" Oct 30 22:16:18.987537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.987496 4335 scope.go:122] "RemoveContainer" containerID="541c4f8f00dff5301f42a0547ccf95b560d883f3bf5cf167d72329ffe244d9a6" Oct 30 22:16:18.997917 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.997902 4335 scope.go:122] "RemoveContainer" containerID="541c4f8f00dff5301f42a0547ccf95b560d883f3bf5cf167d72329ffe244d9a6" Oct 30 22:16:18.998194 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:16:18.998176 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541c4f8f00dff5301f42a0547ccf95b560d883f3bf5cf167d72329ffe244d9a6\": container with ID starting with 541c4f8f00dff5301f42a0547ccf95b560d883f3bf5cf167d72329ffe244d9a6 not found: ID does not exist" containerID="541c4f8f00dff5301f42a0547ccf95b560d883f3bf5cf167d72329ffe244d9a6" Oct 30 22:16:18.998261 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:18.998201 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541c4f8f00dff5301f42a0547ccf95b560d883f3bf5cf167d72329ffe244d9a6"} err="failed to get container status \"541c4f8f00dff5301f42a0547ccf95b560d883f3bf5cf167d72329ffe244d9a6\": rpc error: code = NotFound desc = could not find container \"541c4f8f00dff5301f42a0547ccf95b560d883f3bf5cf167d72329ffe244d9a6\": container with ID starting with 541c4f8f00dff5301f42a0547ccf95b560d883f3bf5cf167d72329ffe244d9a6 not found: ID does not exist" Oct 30 22:16:19.269838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:19.269760 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw"] Oct 30 22:16:19.275277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:19.275231 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-pw5rw"] Oct 30 22:16:19.380102 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:19.380073 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25"] Oct 30 22:16:19.389731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:19.389705 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-6bt25"] Oct 30 22:16:19.499045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:19.499015 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78"] Oct 30 22:16:19.506383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:19.506355 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-c9x78"] Oct 30 22:16:19.611319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:19.611224 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx"] Oct 30 22:16:19.616552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:19.616530 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cef074-openshift-default-569676869-lhszx"] Oct 30 22:16:19.783541 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:19.783509 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4"] Oct 30 22:16:19.783815 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:19.783788 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" podUID="aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" containerName="istio-proxy" containerID="cri-o://79029067a8808fea2a9ce5d661bedd9e7430342afe364406d2794e135bf8ab9e" gracePeriod=30 Oct 30 22:16:19.788290 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:19.788263 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t"] Oct 30 22:16:19.788479 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:19.788459 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" podUID="04c45673-5196-4efe-a65c-fbf6ba5e3c08" containerName="istio-proxy" containerID="cri-o://879c1f553c92b22b3d8dab8a921e5c55748ac388a69d8bc4dcec3b0ff692afa4" gracePeriod=30 Oct 30 22:16:19.792663 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:19.792627 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm"] Oct 30 22:16:19.792860 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:19.792836 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" podUID="a334a2d7-0b91-402c-b690-fff3f5045bfc" containerName="istio-proxy" containerID="cri-o://a30754a4b327226fd220b04950b5f442057d824728224734be3b091487ff6722" gracePeriod=30 Oct 30 22:16:19.796725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:19.796704 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759"] Oct 30 22:16:19.796884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:19.796864 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" podUID="74ffb1a6-5089-419b-a51d-a31f5b1b8649" containerName="istio-proxy" containerID="cri-o://54769dbd213f04f609b772cceb6fb162ae982cc8baabf373896c6a4a4cdaad60" gracePeriod=30 Oct 30 22:16:20.078634 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:20.078592 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7"] Oct 30 22:16:20.078838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:20.078812 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" podUID="87b2369a-d31f-4413-b5c6-ec4b217c3f8b" containerName="istio-proxy" containerID="cri-o://8554f5a3adb87bcc1e5fbb9a1e888c83ca7fa4290ba01ccc487f79862f978309" gracePeriod=30 Oct 30 22:16:20.888721 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:20.888683 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:16:20.889146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:20.888741 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" podUID="a334a2d7-0b91-402c-b690-fff3f5045bfc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:16:21.058861 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:21.058826 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c79ac4e-ddf1-41d0-a8ac-326974828e18" path="/var/lib/kubelet/pods/6c79ac4e-ddf1-41d0-a8ac-326974828e18/volumes" Oct 30 22:16:21.059300 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:21.059285 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" path="/var/lib/kubelet/pods/c55f64be-7022-43a7-a419-cbf1ab06223a/volumes" Oct 30 22:16:21.059753 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:21.059739 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6978eea-f24f-4d30-83b0-85b0015baf53" path="/var/lib/kubelet/pods/d6978eea-f24f-4d30-83b0-85b0015baf53/volumes" Oct 30 22:16:21.060483 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:21.060466 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" path="/var/lib/kubelet/pods/e372cc9b-a3ca-4ad2-975d-cdaa392a87d1/volumes" Oct 30 22:16:23.034150 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:23.034113 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:16:25.093288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.093200 4335 generic.go:356] "Generic (PLEG): container finished" podID="a334a2d7-0b91-402c-b690-fff3f5045bfc" containerID="a30754a4b327226fd220b04950b5f442057d824728224734be3b091487ff6722" exitCode=0 Oct 30 22:16:25.124637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.123335 4335 generic.go:356] "Generic (PLEG): container finished" podID="aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" containerID="79029067a8808fea2a9ce5d661bedd9e7430342afe364406d2794e135bf8ab9e" exitCode=0 Oct 30 22:16:25.223640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.183687 4335 generic.go:356] "Generic (PLEG): container finished" podID="04c45673-5196-4efe-a65c-fbf6ba5e3c08" containerID="879c1f553c92b22b3d8dab8a921e5c55748ac388a69d8bc4dcec3b0ff692afa4" exitCode=0 Oct 30 22:16:25.223640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.187271 4335 generic.go:356] "Generic (PLEG): container finished" podID="74ffb1a6-5089-419b-a51d-a31f5b1b8649" containerID="54769dbd213f04f609b772cceb6fb162ae982cc8baabf373896c6a4a4cdaad60" exitCode=0 Oct 30 22:16:25.244685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.241586 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" event={"ID":"a334a2d7-0b91-402c-b690-fff3f5045bfc","Type":"ContainerDied","Data":"a30754a4b327226fd220b04950b5f442057d824728224734be3b091487ff6722"} Oct 30 22:16:25.244685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.241680 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" event={"ID":"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1","Type":"ContainerDied","Data":"79029067a8808fea2a9ce5d661bedd9e7430342afe364406d2794e135bf8ab9e"} Oct 30 22:16:25.244685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.241719 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" event={"ID":"04c45673-5196-4efe-a65c-fbf6ba5e3c08","Type":"ContainerDied","Data":"879c1f553c92b22b3d8dab8a921e5c55748ac388a69d8bc4dcec3b0ff692afa4"} Oct 30 22:16:25.244685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.241749 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" event={"ID":"74ffb1a6-5089-419b-a51d-a31f5b1b8649","Type":"ContainerDied","Data":"54769dbd213f04f609b772cceb6fb162ae982cc8baabf373896c6a4a4cdaad60"} Oct 30 22:16:25.244685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.241784 4335 scope.go:122] "RemoveContainer" containerID="03850171021aacd90821ddd621b0e1ea020bcd503e73f6ffd955d922de2635b8" Oct 30 22:16:25.471357 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:16:25.471325 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:16:25.489542 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.489030 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 22:16:25.591101 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.591070 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-workload-certs\") pod \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " Oct 30 22:16:25.591288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.591168 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-envoy\") pod \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " Oct 30 22:16:25.591288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.591204 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-data\") pod \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " Oct 30 22:16:25.591288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.591245 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-credential-socket\") pod \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " Oct 30 22:16:25.591288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.591272 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-workload-socket\") pod \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " Oct 30 22:16:25.591509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.591361 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-token\" (UniqueName: \"kubernetes.io/projected/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-token\") pod \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " Oct 30 22:16:25.591509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.591399 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-kube-api-access-z2ghc\" (UniqueName: \"kubernetes.io/projected/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-kube-api-access-z2ghc\") pod \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " Oct 30 22:16:25.591509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.591428 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-podinfo\") pod \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " Oct 30 22:16:25.591742 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.591589 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istiod-ca-cert\") pod \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\" (UID: \"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1\") " Oct 30 22:16:25.662734 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.662389 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-workload-socket" pod "aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" (UID: "aa8ee860-7c1a-43f1-8e21-634a5bc0dee1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:25.664001 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.663041 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-credential-socket" pod "aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" (UID: "aa8ee860-7c1a-43f1-8e21-634a5bc0dee1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:25.671442 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.664297 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istiod-ca-cert" pod "aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" (UID: "aa8ee860-7c1a-43f1-8e21-634a5bc0dee1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:25.678081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.677688 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-data" pod "aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" (UID: "aa8ee860-7c1a-43f1-8e21-634a5bc0dee1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:25.678081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.677768 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-workload-certs" pod "aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" (UID: "aa8ee860-7c1a-43f1-8e21-634a5bc0dee1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:25.708640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.707968 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:25.708640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.708007 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:25.708640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.708024 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:25.708640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.708040 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:25.708640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.708055 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:25.836018 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.829179 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-podinfo" pod "aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" (UID: "aa8ee860-7c1a-43f1-8e21-634a5bc0dee1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:25.899637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:25.851391 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.013668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.012781 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-token" pod "aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" (UID: "aa8ee860-7c1a-43f1-8e21-634a5bc0dee1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:26.014705 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.014676 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.014828 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.014784 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-envoy" pod "aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" (UID: "aa8ee860-7c1a-43f1-8e21-634a5bc0dee1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.019123 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.017240 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-kube-api-access-z2ghc" pod "aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" (UID: "aa8ee860-7c1a-43f1-8e21-634a5bc0dee1"). InnerVolumeSpecName "kube-api-access-z2ghc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:26.100243 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.099531 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:16:26.103314 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.103174 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:16:26.127832 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.127364 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.127832 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.127398 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2ghc\" (UniqueName: \"kubernetes.io/projected/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1-kube-api-access-z2ghc\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.163834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.163787 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:16:26.210289 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.210189 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" event={"ID":"aa8ee860-7c1a-43f1-8e21-634a5bc0dee1","Type":"ContainerDied","Data":"9074ab260ad5c3b7e55dade916d6ce00c4b1420401b7d3872057c0e3b6119ed8"} Oct 30 22:16:26.210533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.210509 4335 scope.go:122] "RemoveContainer" containerID="79029067a8808fea2a9ce5d661bedd9e7430342afe364406d2794e135bf8ab9e" Oct 30 22:16:26.210831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.210818 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4" Oct 30 22:16:26.215533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.215514 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" Oct 30 22:16:26.215609 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.215517 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t" event={"ID":"04c45673-5196-4efe-a65c-fbf6ba5e3c08","Type":"ContainerDied","Data":"20bbf9df512b85e907c9098f62402d25696f3576de39314794e11885cf1c517a"} Oct 30 22:16:26.218186 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.217667 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" Oct 30 22:16:26.218186 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.217801 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759" event={"ID":"74ffb1a6-5089-419b-a51d-a31f5b1b8649","Type":"ContainerDied","Data":"2c3991abbdb288d38d4ed701e9ab4cce9032d3eab3eacf80acfd326c7a5ed7e2"} Oct 30 22:16:26.220538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.220357 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" Oct 30 22:16:26.220538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.220395 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm" event={"ID":"a334a2d7-0b91-402c-b690-fff3f5045bfc","Type":"ContainerDied","Data":"c34149db5347c66d61c16a18530c3415ddb20762e5953f53b44ee2033dcfca67"} Oct 30 22:16:26.222688 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.222563 4335 generic.go:356] "Generic (PLEG): container finished" podID="87b2369a-d31f-4413-b5c6-ec4b217c3f8b" containerID="8554f5a3adb87bcc1e5fbb9a1e888c83ca7fa4290ba01ccc487f79862f978309" exitCode=0 Oct 30 22:16:26.222688 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.222655 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" event={"ID":"87b2369a-d31f-4413-b5c6-ec4b217c3f8b","Type":"ContainerDied","Data":"8554f5a3adb87bcc1e5fbb9a1e888c83ca7fa4290ba01ccc487f79862f978309"} Oct 30 22:16:26.230498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.230473 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-credential-socket\") pod \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " Oct 30 22:16:26.230596 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.230528 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-envoy\") pod \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " Oct 30 22:16:26.230596 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.230579 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-data\") pod \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " Oct 30 22:16:26.230742 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.230661 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-podinfo\") pod \"a334a2d7-0b91-402c-b690-fff3f5045bfc\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " Oct 30 22:16:26.230742 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.230693 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-credential-socket\") pod \"a334a2d7-0b91-402c-b690-fff3f5045bfc\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " Oct 30 22:16:26.230742 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.230729 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-token\" (UniqueName: \"kubernetes.io/projected/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-token\") pod \"a334a2d7-0b91-402c-b690-fff3f5045bfc\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " Oct 30 22:16:26.230892 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.230812 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-data\") pod \"a334a2d7-0b91-402c-b690-fff3f5045bfc\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " Oct 30 22:16:26.230892 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.230855 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-workload-certs\") pod \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " Oct 30 22:16:26.230996 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.230926 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a334a2d7-0b91-402c-b690-fff3f5045bfc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a334a2d7-0b91-402c-b690-fff3f5045bfc-istiod-ca-cert\") pod \"a334a2d7-0b91-402c-b690-fff3f5045bfc\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " Oct 30 22:16:26.230996 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.230956 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-token\" (UniqueName: \"kubernetes.io/projected/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-token\") pod \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " Oct 30 22:16:26.231109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.230992 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istiod-ca-cert\") pod \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " Oct 30 22:16:26.231109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.231053 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-envoy\") pod \"a334a2d7-0b91-402c-b690-fff3f5045bfc\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " Oct 30 22:16:26.231109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.231082 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a334a2d7-0b91-402c-b690-fff3f5045bfc-kube-api-access-ks6hg\" (UniqueName: \"kubernetes.io/projected/a334a2d7-0b91-402c-b690-fff3f5045bfc-kube-api-access-ks6hg\") pod \"a334a2d7-0b91-402c-b690-fff3f5045bfc\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " Oct 30 22:16:26.231252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.231123 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-token\" (UniqueName: \"kubernetes.io/projected/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-token\") pod \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " Oct 30 22:16:26.231252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.231161 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-workload-certs\") pod \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " Oct 30 22:16:26.231252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.231197 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-data\") pod \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " Oct 30 22:16:26.231406 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.231307 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c45673-5196-4efe-a65c-fbf6ba5e3c08-kube-api-access-7cv2d\" (UniqueName: \"kubernetes.io/projected/04c45673-5196-4efe-a65c-fbf6ba5e3c08-kube-api-access-7cv2d\") pod \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " Oct 30 22:16:26.231406 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.231343 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-podinfo\") pod \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " Oct 30 22:16:26.231406 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.231386 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istiod-ca-cert\") pod \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " Oct 30 22:16:26.231558 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.231447 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-workload-socket\") pod \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " Oct 30 22:16:26.231558 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.231510 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-workload-certs\") pod \"a334a2d7-0b91-402c-b690-fff3f5045bfc\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " Oct 30 22:16:26.231702 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.231562 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-workload-socket\") pod \"a334a2d7-0b91-402c-b690-fff3f5045bfc\" (UID: \"a334a2d7-0b91-402c-b690-fff3f5045bfc\") " Oct 30 22:16:26.231702 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.231591 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-workload-socket\") pod \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " Oct 30 22:16:26.232112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.232092 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-podinfo\") pod \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " Oct 30 22:16:26.232193 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.232163 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-envoy\") pod \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " Oct 30 22:16:26.232253 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.232234 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-credential-socket\") pod \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\" (UID: \"04c45673-5196-4efe-a65c-fbf6ba5e3c08\") " Oct 30 22:16:26.232306 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.232287 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ffb1a6-5089-419b-a51d-a31f5b1b8649-kube-api-access-ggjkq\" (UniqueName: \"kubernetes.io/projected/74ffb1a6-5089-419b-a51d-a31f5b1b8649-kube-api-access-ggjkq\") pod \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\" (UID: \"74ffb1a6-5089-419b-a51d-a31f5b1b8649\") " Oct 30 22:16:26.233120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.233035 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a334a2d7-0b91-402c-b690-fff3f5045bfc-istiod-ca-cert" pod "a334a2d7-0b91-402c-b690-fff3f5045bfc" (UID: "a334a2d7-0b91-402c-b690-fff3f5045bfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:26.233716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.233483 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-data" pod "a334a2d7-0b91-402c-b690-fff3f5045bfc" (UID: "a334a2d7-0b91-402c-b690-fff3f5045bfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.234037 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.233862 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-workload-certs" pod "74ffb1a6-5089-419b-a51d-a31f5b1b8649" (UID: "74ffb1a6-5089-419b-a51d-a31f5b1b8649"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.234331 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.234164 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-data" pod "04c45673-5196-4efe-a65c-fbf6ba5e3c08" (UID: "04c45673-5196-4efe-a65c-fbf6ba5e3c08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.234740 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.234463 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-credential-socket" pod "74ffb1a6-5089-419b-a51d-a31f5b1b8649" (UID: "74ffb1a6-5089-419b-a51d-a31f5b1b8649"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.261728 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.257251 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-credential-socket" pod "a334a2d7-0b91-402c-b690-fff3f5045bfc" (UID: "a334a2d7-0b91-402c-b690-fff3f5045bfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.261728 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.257259 4335 scope.go:122] "RemoveContainer" containerID="879c1f553c92b22b3d8dab8a921e5c55748ac388a69d8bc4dcec3b0ff692afa4" Oct 30 22:16:26.261728 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.257481 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istiod-ca-cert" pod "74ffb1a6-5089-419b-a51d-a31f5b1b8649" (UID: "74ffb1a6-5089-419b-a51d-a31f5b1b8649"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:26.261728 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.257856 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-workload-socket" pod "74ffb1a6-5089-419b-a51d-a31f5b1b8649" (UID: "74ffb1a6-5089-419b-a51d-a31f5b1b8649"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.261728 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.258046 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-workload-certs" pod "04c45673-5196-4efe-a65c-fbf6ba5e3c08" (UID: "04c45673-5196-4efe-a65c-fbf6ba5e3c08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.261728 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.258227 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-data" pod "74ffb1a6-5089-419b-a51d-a31f5b1b8649" (UID: "74ffb1a6-5089-419b-a51d-a31f5b1b8649"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.261728 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.258388 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a334a2d7-0b91-402c-b690-fff3f5045bfc-kube-api-access-ks6hg" pod "a334a2d7-0b91-402c-b690-fff3f5045bfc" (UID: "a334a2d7-0b91-402c-b690-fff3f5045bfc"). InnerVolumeSpecName "kube-api-access-ks6hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:26.261728 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.261332 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istiod-ca-cert" pod "04c45673-5196-4efe-a65c-fbf6ba5e3c08" (UID: "04c45673-5196-4efe-a65c-fbf6ba5e3c08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:26.261728 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.261596 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-workload-socket" pod "04c45673-5196-4efe-a65c-fbf6ba5e3c08" (UID: "04c45673-5196-4efe-a65c-fbf6ba5e3c08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.262385 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.261795 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-workload-certs" pod "a334a2d7-0b91-402c-b690-fff3f5045bfc" (UID: "a334a2d7-0b91-402c-b690-fff3f5045bfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.262385 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.261996 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-workload-socket" pod "a334a2d7-0b91-402c-b690-fff3f5045bfc" (UID: "a334a2d7-0b91-402c-b690-fff3f5045bfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.264325 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.263501 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-credential-socket" pod "04c45673-5196-4efe-a65c-fbf6ba5e3c08" (UID: "04c45673-5196-4efe-a65c-fbf6ba5e3c08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.270656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.269960 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-envoy" pod "04c45673-5196-4efe-a65c-fbf6ba5e3c08" (UID: "04c45673-5196-4efe-a65c-fbf6ba5e3c08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.270656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.270070 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-token" pod "04c45673-5196-4efe-a65c-fbf6ba5e3c08" (UID: "04c45673-5196-4efe-a65c-fbf6ba5e3c08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:26.272430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.272392 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-podinfo" pod "04c45673-5196-4efe-a65c-fbf6ba5e3c08" (UID: "04c45673-5196-4efe-a65c-fbf6ba5e3c08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:26.273469 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.273442 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-podinfo" pod "a334a2d7-0b91-402c-b690-fff3f5045bfc" (UID: "a334a2d7-0b91-402c-b690-fff3f5045bfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:26.274437 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.274362 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:16:26.274539 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.274466 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-envoy" pod "a334a2d7-0b91-402c-b690-fff3f5045bfc" (UID: "a334a2d7-0b91-402c-b690-fff3f5045bfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.276966 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.276934 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-token" pod "74ffb1a6-5089-419b-a51d-a31f5b1b8649" (UID: "74ffb1a6-5089-419b-a51d-a31f5b1b8649"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:26.279192 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.279153 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c45673-5196-4efe-a65c-fbf6ba5e3c08-kube-api-access-7cv2d" pod "04c45673-5196-4efe-a65c-fbf6ba5e3c08" (UID: "04c45673-5196-4efe-a65c-fbf6ba5e3c08"). InnerVolumeSpecName "kube-api-access-7cv2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:26.282896 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.282879 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-envoy" pod "74ffb1a6-5089-419b-a51d-a31f5b1b8649" (UID: "74ffb1a6-5089-419b-a51d-a31f5b1b8649"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.284147 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.283962 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-podinfo" pod "74ffb1a6-5089-419b-a51d-a31f5b1b8649" (UID: "74ffb1a6-5089-419b-a51d-a31f5b1b8649"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:26.290313 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.290276 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ffb1a6-5089-419b-a51d-a31f5b1b8649-kube-api-access-ggjkq" pod "74ffb1a6-5089-419b-a51d-a31f5b1b8649" (UID: "74ffb1a6-5089-419b-a51d-a31f5b1b8649"). InnerVolumeSpecName "kube-api-access-ggjkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:26.291126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.291101 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-token" pod "a334a2d7-0b91-402c-b690-fff3f5045bfc" (UID: "a334a2d7-0b91-402c-b690-fff3f5045bfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:26.293285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.293263 4335 scope.go:122] "RemoveContainer" containerID="54769dbd213f04f609b772cceb6fb162ae982cc8baabf373896c6a4a4cdaad60" Oct 30 22:16:26.321901 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.318868 4335 scope.go:122] "RemoveContainer" containerID="a30754a4b327226fd220b04950b5f442057d824728224734be3b091487ff6722" Oct 30 22:16:26.336707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.335128 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-kube-api-access-mhhp5\" (UniqueName: \"kubernetes.io/projected/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-kube-api-access-mhhp5\") pod \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " Oct 30 22:16:26.336707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.335192 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-envoy\") pod \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " Oct 30 22:16:26.336707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.335302 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-podinfo\") pod \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " Oct 30 22:16:26.336707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.335341 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-workload-certs\") pod \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " Oct 30 22:16:26.336707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.335390 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-token\" (UniqueName: \"kubernetes.io/projected/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-token\") pod \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " Oct 30 22:16:26.336707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.335504 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istiod-ca-cert\") pod \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " Oct 30 22:16:26.336707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.335578 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-credential-socket\") pod \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " Oct 30 22:16:26.336707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.335751 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-data\") pod \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " Oct 30 22:16:26.336707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.335809 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-workload-socket\") pod \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\" (UID: \"87b2369a-d31f-4413-b5c6-ec4b217c3f8b\") " Oct 30 22:16:26.336707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.336470 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.336707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.336494 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.336707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.336510 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggjkq\" (UniqueName: \"kubernetes.io/projected/74ffb1a6-5089-419b-a51d-a31f5b1b8649-kube-api-access-ggjkq\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.336707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.336528 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.336707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.336543 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.336707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.336557 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.336707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.336572 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.336707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.336587 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.336707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.336601 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.338344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.337887 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.338344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.337916 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.338344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.337931 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a334a2d7-0b91-402c-b690-fff3f5045bfc-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.338344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.337948 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.338344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.337962 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.338344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.337978 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.338344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.337994 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks6hg\" (UniqueName: \"kubernetes.io/projected/a334a2d7-0b91-402c-b690-fff3f5045bfc-kube-api-access-ks6hg\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.338344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.338009 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.338344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.338025 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.338344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.338039 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.338344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.338055 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cv2d\" (UniqueName: \"kubernetes.io/projected/04c45673-5196-4efe-a65c-fbf6ba5e3c08-kube-api-access-7cv2d\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.338344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.338070 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ffb1a6-5089-419b-a51d-a31f5b1b8649-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.338344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.338084 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.338344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.338097 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ffb1a6-5089-419b-a51d-a31f5b1b8649-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.338344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.338110 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.338344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.338125 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a334a2d7-0b91-402c-b690-fff3f5045bfc-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.338344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.338139 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c45673-5196-4efe-a65c-fbf6ba5e3c08-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.338344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.338153 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c45673-5196-4efe-a65c-fbf6ba5e3c08-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.339788 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.339458 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-workload-socket" pod "87b2369a-d31f-4413-b5c6-ec4b217c3f8b" (UID: "87b2369a-d31f-4413-b5c6-ec4b217c3f8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.351831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.350676 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-credential-socket" pod "87b2369a-d31f-4413-b5c6-ec4b217c3f8b" (UID: "87b2369a-d31f-4413-b5c6-ec4b217c3f8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.351831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.350711 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-workload-certs" pod "87b2369a-d31f-4413-b5c6-ec4b217c3f8b" (UID: "87b2369a-d31f-4413-b5c6-ec4b217c3f8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.351831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.351096 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istiod-ca-cert" pod "87b2369a-d31f-4413-b5c6-ec4b217c3f8b" (UID: "87b2369a-d31f-4413-b5c6-ec4b217c3f8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:26.351831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.351795 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-data" pod "87b2369a-d31f-4413-b5c6-ec4b217c3f8b" (UID: "87b2369a-d31f-4413-b5c6-ec4b217c3f8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.354822 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.354786 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-envoy" pod "87b2369a-d31f-4413-b5c6-ec4b217c3f8b" (UID: "87b2369a-d31f-4413-b5c6-ec4b217c3f8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:26.357012 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.356978 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-token" pod "87b2369a-d31f-4413-b5c6-ec4b217c3f8b" (UID: "87b2369a-d31f-4413-b5c6-ec4b217c3f8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:26.357519 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.357494 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-podinfo" pod "87b2369a-d31f-4413-b5c6-ec4b217c3f8b" (UID: "87b2369a-d31f-4413-b5c6-ec4b217c3f8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:26.360161 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.360124 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-kube-api-access-mhhp5" pod "87b2369a-d31f-4413-b5c6-ec4b217c3f8b" (UID: "87b2369a-d31f-4413-b5c6-ec4b217c3f8b"). InnerVolumeSpecName "kube-api-access-mhhp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:26.441429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.439645 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.441429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.439681 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.441429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.439698 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhhp5\" (UniqueName: \"kubernetes.io/projected/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-kube-api-access-mhhp5\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.441429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.439711 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.441429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.439725 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.441429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.439740 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.441429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.439754 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.441429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.439767 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.441429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.439781 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b2369a-d31f-4413-b5c6-ec4b217c3f8b-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:26.553826 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.553794 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4"] Oct 30 22:16:26.560032 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.560007 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-h9kl4"] Oct 30 22:16:26.791770 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.791677 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm"] Oct 30 22:16:26.800167 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:26.800137 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-tp7zm"] Oct 30 22:16:27.116095 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:27.116055 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a334a2d7-0b91-402c-b690-fff3f5045bfc" path="/var/lib/kubelet/pods/a334a2d7-0b91-402c-b690-fff3f5045bfc/volumes" Oct 30 22:16:27.127837 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:27.127805 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" path="/var/lib/kubelet/pods/aa8ee860-7c1a-43f1-8e21-634a5bc0dee1/volumes" Oct 30 22:16:27.139714 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:27.135705 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759"] Oct 30 22:16:27.139990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:27.139968 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-p5759"] Oct 30 22:16:27.209668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:27.198189 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t"] Oct 30 22:16:27.209668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:27.199360 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-2g68t"] Oct 30 22:16:27.248112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:27.248071 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" event={"ID":"87b2369a-d31f-4413-b5c6-ec4b217c3f8b","Type":"ContainerDied","Data":"dc52df387192676062840a9952066c80ef74d1b37d55f395b1bf2bfde1da0894"} Oct 30 22:16:27.248309 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:27.248129 4335 scope.go:122] "RemoveContainer" containerID="8554f5a3adb87bcc1e5fbb9a1e888c83ca7fa4290ba01ccc487f79862f978309" Oct 30 22:16:27.248309 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:27.248250 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7" Oct 30 22:16:27.419768 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:27.417401 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7"] Oct 30 22:16:27.424555 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:27.424521 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-lsnz7"] Oct 30 22:16:28.746089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:28.746060 4335 scope.go:122] "RemoveContainer" containerID="b5aac2395adde8b737f884f46ea6e4c8bce9a1ac4e4526ce18b819ba1d94a997" Oct 30 22:16:28.756999 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:28.756980 4335 scope.go:122] "RemoveContainer" containerID="609ca12fc43812ddb27cd394d798b266456b1dedf14c6b76a291fd4b5b17c3e3" Oct 30 22:16:28.784247 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:28.784224 4335 scope.go:122] "RemoveContainer" containerID="8b349a34e4d208080d13b79622da8e84dd519cbd8339606fbd258d738f63082c" Oct 30 22:16:28.804000 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:28.802896 4335 scope.go:122] "RemoveContainer" containerID="a49bee525deea056c69c768d3523176701c6a17e31d098dc1bf4996e8ff3de2e" Oct 30 22:16:28.815440 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:28.815416 4335 scope.go:122] "RemoveContainer" containerID="547ac3bfb5028c2ded88a3a43540e9d8e32abd32d30b9dcf364d928d761ed231" Oct 30 22:16:28.828264 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:28.828232 4335 scope.go:122] "RemoveContainer" containerID="fc0070fcdc0ab30b5518df27987e35ee1b7e7b7891a4595f65d7393fd52e88a0" Oct 30 22:16:28.840415 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:28.840400 4335 scope.go:122] "RemoveContainer" containerID="674050046b40ae24c4a90dc8a64e50663b7ae4e73c9d5bd02ebbe968abc149d3" Oct 30 22:16:28.851476 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:28.851450 4335 scope.go:122] "RemoveContainer" containerID="ff1dfbab03d60cd1377d7b9b7dca4c29fdd74c24eeefc614259ddeb6af3f3375" Oct 30 22:16:28.861156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:28.861132 4335 scope.go:122] "RemoveContainer" containerID="22bbc2228ae9223eaa5a1fc913593d026d49e8cc20b3f75170049ca320e5c7d0" Oct 30 22:16:29.061363 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:29.061332 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04c45673-5196-4efe-a65c-fbf6ba5e3c08" path="/var/lib/kubelet/pods/04c45673-5196-4efe-a65c-fbf6ba5e3c08/volumes" Oct 30 22:16:29.061956 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:29.061938 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ffb1a6-5089-419b-a51d-a31f5b1b8649" path="/var/lib/kubelet/pods/74ffb1a6-5089-419b-a51d-a31f5b1b8649/volumes" Oct 30 22:16:29.062434 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:29.062414 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87b2369a-d31f-4413-b5c6-ec4b217c3f8b" path="/var/lib/kubelet/pods/87b2369a-d31f-4413-b5c6-ec4b217c3f8b/volumes" Oct 30 22:16:35.024191 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:16:35.024143 4335 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice/crio-dc52df387192676062840a9952066c80ef74d1b37d55f395b1bf2bfde1da0894\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice\": RecentStats: unable to find data in memory cache]" Oct 30 22:16:36.102905 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:16:36.102864 4335 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice/crio-dc52df387192676062840a9952066c80ef74d1b37d55f395b1bf2bfde1da0894\": RecentStats: unable to find data in memory cache]" Oct 30 22:16:43.677154 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677079 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7"] Oct 30 22:16:43.677826 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677809 4335 cpu_manager.go:414] "containerMap: removing container" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerName="istio-proxy" Oct 30 22:16:43.677928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677827 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerName="istio-proxy" Oct 30 22:16:43.677928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677839 4335 cpu_manager.go:414] "containerMap: removing container" podUID="aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" containerName="istio-proxy" Oct 30 22:16:43.677928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677844 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" containerName="istio-proxy" Oct 30 22:16:43.677928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677854 4335 cpu_manager.go:414] "containerMap: removing container" podUID="6c79ac4e-ddf1-41d0-a8ac-326974828e18" containerName="istio-proxy" Oct 30 22:16:43.677928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677860 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c79ac4e-ddf1-41d0-a8ac-326974828e18" containerName="istio-proxy" Oct 30 22:16:43.677928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677873 4335 cpu_manager.go:414] "containerMap: removing container" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerName="istio-proxy" Oct 30 22:16:43.677928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677878 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerName="istio-proxy" Oct 30 22:16:43.677928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677889 4335 cpu_manager.go:414] "containerMap: removing container" podUID="87b2369a-d31f-4413-b5c6-ec4b217c3f8b" containerName="istio-proxy" Oct 30 22:16:43.677928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677894 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b2369a-d31f-4413-b5c6-ec4b217c3f8b" containerName="istio-proxy" Oct 30 22:16:43.677928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677902 4335 cpu_manager.go:414] "containerMap: removing container" podUID="74ffb1a6-5089-419b-a51d-a31f5b1b8649" containerName="istio-proxy" Oct 30 22:16:43.677928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677908 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ffb1a6-5089-419b-a51d-a31f5b1b8649" containerName="istio-proxy" Oct 30 22:16:43.677928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677919 4335 cpu_manager.go:414] "containerMap: removing container" podUID="44c4a62d-b9e3-49a2-9f30-02197204f81e" containerName="revoke-keys" Oct 30 22:16:43.677928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677925 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c4a62d-b9e3-49a2-9f30-02197204f81e" containerName="revoke-keys" Oct 30 22:16:43.678319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677952 4335 cpu_manager.go:414] "containerMap: removing container" podUID="a334a2d7-0b91-402c-b690-fff3f5045bfc" containerName="istio-proxy" Oct 30 22:16:43.678319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677957 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a334a2d7-0b91-402c-b690-fff3f5045bfc" containerName="istio-proxy" Oct 30 22:16:43.678319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677966 4335 cpu_manager.go:414] "containerMap: removing container" podUID="d6978eea-f24f-4d30-83b0-85b0015baf53" containerName="istio-proxy" Oct 30 22:16:43.678319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677971 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6978eea-f24f-4d30-83b0-85b0015baf53" containerName="istio-proxy" Oct 30 22:16:43.678319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677979 4335 cpu_manager.go:414] "containerMap: removing container" podUID="04c45673-5196-4efe-a65c-fbf6ba5e3c08" containerName="istio-proxy" Oct 30 22:16:43.678319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.677983 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c45673-5196-4efe-a65c-fbf6ba5e3c08" containerName="istio-proxy" Oct 30 22:16:43.678319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.678085 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="a334a2d7-0b91-402c-b690-fff3f5045bfc" containerName="istio-proxy" Oct 30 22:16:43.678319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.678095 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" containerName="istio-proxy" Oct 30 22:16:43.678319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.678101 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6978eea-f24f-4d30-83b0-85b0015baf53" containerName="istio-proxy" Oct 30 22:16:43.678319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.678109 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c45673-5196-4efe-a65c-fbf6ba5e3c08" containerName="istio-proxy" Oct 30 22:16:43.678319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.678114 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="e372cc9b-a3ca-4ad2-975d-cdaa392a87d1" containerName="istio-proxy" Oct 30 22:16:43.678319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.678122 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" containerName="istio-proxy" Oct 30 22:16:43.678319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.678128 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c79ac4e-ddf1-41d0-a8ac-326974828e18" containerName="istio-proxy" Oct 30 22:16:43.678319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.678135 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="c55f64be-7022-43a7-a419-cbf1ab06223a" containerName="istio-proxy" Oct 30 22:16:43.678319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.678142 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="87b2369a-d31f-4413-b5c6-ec4b217c3f8b" containerName="istio-proxy" Oct 30 22:16:43.678319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.678148 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ffb1a6-5089-419b-a51d-a31f5b1b8649" containerName="istio-proxy" Oct 30 22:16:43.678319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.678156 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c4a62d-b9e3-49a2-9f30-02197204f81e" containerName="revoke-keys" Oct 30 22:16:43.685872 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.685840 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7" Oct 30 22:16:43.739913 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.739878 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7"] Oct 30 22:16:43.786149 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.782344 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-b8a091\"" Oct 30 22:16:43.786149 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.783119 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-b8a091-dockercfg-f4vvs\"" Oct 30 22:16:43.825679 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.825631 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhhhn\" (UniqueName: \"kubernetes.io/projected/8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1-kube-api-access-lhhhn\") pod \"payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7\" (UID: \"8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7" Oct 30 22:16:43.825915 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.825895 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7\" (UID: \"8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7" Oct 30 22:16:43.927291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.927198 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhhhn\" (UniqueName: \"kubernetes.io/projected/8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1-kube-api-access-lhhhn\") pod \"payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7\" (UID: \"8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7" Oct 30 22:16:43.927464 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.927448 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7\" (UID: \"8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7" Oct 30 22:16:43.928384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.928358 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7\" (UID: \"8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7" Oct 30 22:16:43.989920 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:43.989891 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhhhn\" (UniqueName: \"kubernetes.io/projected/8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1-kube-api-access-lhhhn\") pod \"payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7\" (UID: \"8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7" Oct 30 22:16:44.002667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:44.002635 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7" Oct 30 22:16:44.443382 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:44.443352 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7"] Oct 30 22:16:44.443966 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:16:44.443940 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbff4d0_50fb_48f1_bfe6_fa97318fa8a1.slice/crio-7cb0ea0a7fa6ff7af10ee6072e8fe33c5b0caf786ac61f261f7e8b18a4f4eb55 WatchSource:0}: Error finding container 7cb0ea0a7fa6ff7af10ee6072e8fe33c5b0caf786ac61f261f7e8b18a4f4eb55: Status 404 returned error can't find the container with id 7cb0ea0a7fa6ff7af10ee6072e8fe33c5b0caf786ac61f261f7e8b18a4f4eb55 Oct 30 22:16:44.466183 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:44.466155 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7" event={"ID":"8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1","Type":"ContainerStarted","Data":"7cb0ea0a7fa6ff7af10ee6072e8fe33c5b0caf786ac61f261f7e8b18a4f4eb55"} Oct 30 22:16:45.477206 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:45.477173 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7" event={"ID":"8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1","Type":"ContainerStarted","Data":"f26c219bc75ac5ad8acc006319fff4b0a6e5e0708fe481a64e6a18f767a72bd6"} Oct 30 22:16:45.477717 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:45.477234 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7" Oct 30 22:16:45.608198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:45.608149 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7" podStartSLOduration=2.608135568 podStartE2EDuration="2.608135568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:45.602808753 +0000 UTC m=+1938.703240496" watchObservedRunningTime="2026-10-30 22:16:45.608135568 +0000 UTC m=+1938.708567300" Oct 30 22:16:46.192341 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:16:46.192298 4335 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice/crio-dc52df387192676062840a9952066c80ef74d1b37d55f395b1bf2bfde1da0894\": RecentStats: unable to find data in memory cache]" Oct 30 22:16:49.787602 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:16:49.787566 4335 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice/crio-dc52df387192676062840a9952066c80ef74d1b37d55f395b1bf2bfde1da0894\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice\": RecentStats: unable to find data in memory cache]" Oct 30 22:16:55.322266 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:16:55.322217 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:16:56.276602 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:16:56.276572 4335 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice/crio-dc52df387192676062840a9952066c80ef74d1b37d55f395b1bf2bfde1da0894\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice\": RecentStats: unable to find data in memory cache]" Oct 30 22:16:56.490508 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:56.490485 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7" Oct 30 22:16:58.537017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:58.536983 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2"] Oct 30 22:16:58.537515 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:58.537255 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" podUID="88f88e61-2539-422d-b204-cf54cec014f3" containerName="istio-proxy" containerID="cri-o://b4c977a14769edffa5b840b1b35a81ecdc750bbfc3c7c65deb1f28b97f134d66" gracePeriod=30 Oct 30 22:16:58.585571 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:58.585543 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m"] Oct 30 22:16:58.585766 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:58.585745 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" podUID="769e08d0-bf5e-404e-8b7a-adf43dcd0905" containerName="istio-proxy" containerID="cri-o://18ba68e9205228fe0bf3ac3622860c1cd4bbd08d20f61b64ff632720e97009e1" gracePeriod=30 Oct 30 22:16:59.381099 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:16:59.381054 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:17:03.893582 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.893555 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:17:03.925280 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.925255 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:17:03.994533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.994497 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88f88e61-2539-422d-b204-cf54cec014f3-istio-token\" (UniqueName: \"kubernetes.io/projected/88f88e61-2539-422d-b204-cf54cec014f3-istio-token\") pod \"88f88e61-2539-422d-b204-cf54cec014f3\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " Oct 30 22:17:03.994734 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.994601 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-istio-data\") pod \"88f88e61-2539-422d-b204-cf54cec014f3\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " Oct 30 22:17:03.994734 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.994692 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-istio-envoy\") pod \"88f88e61-2539-422d-b204-cf54cec014f3\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " Oct 30 22:17:03.994734 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.994730 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-workload-certs\") pod \"88f88e61-2539-422d-b204-cf54cec014f3\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " Oct 30 22:17:03.994912 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.994759 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88f88e61-2539-422d-b204-cf54cec014f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f88e61-2539-422d-b204-cf54cec014f3-istio-podinfo\") pod \"88f88e61-2539-422d-b204-cf54cec014f3\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " Oct 30 22:17:03.994912 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.994840 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-istio-data" pod "88f88e61-2539-422d-b204-cf54cec014f3" (UID: "88f88e61-2539-422d-b204-cf54cec014f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:03.994912 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.994889 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88f88e61-2539-422d-b204-cf54cec014f3-kube-api-access-nwcjp\" (UniqueName: \"kubernetes.io/projected/88f88e61-2539-422d-b204-cf54cec014f3-kube-api-access-nwcjp\") pod \"88f88e61-2539-422d-b204-cf54cec014f3\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " Oct 30 22:17:03.995081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.994920 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88f88e61-2539-422d-b204-cf54cec014f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f88e61-2539-422d-b204-cf54cec014f3-istiod-ca-cert\") pod \"88f88e61-2539-422d-b204-cf54cec014f3\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " Oct 30 22:17:03.995081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.995047 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-workload-certs" pod "88f88e61-2539-422d-b204-cf54cec014f3" (UID: "88f88e61-2539-422d-b204-cf54cec014f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:03.995218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.995194 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-workload-socket\") pod \"88f88e61-2539-422d-b204-cf54cec014f3\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " Oct 30 22:17:03.995284 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.995252 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88f88e61-2539-422d-b204-cf54cec014f3-istiod-ca-cert" pod "88f88e61-2539-422d-b204-cf54cec014f3" (UID: "88f88e61-2539-422d-b204-cf54cec014f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:03.995336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.995288 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-credential-socket\") pod \"88f88e61-2539-422d-b204-cf54cec014f3\" (UID: \"88f88e61-2539-422d-b204-cf54cec014f3\") " Oct 30 22:17:03.995548 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.995523 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-workload-socket" pod "88f88e61-2539-422d-b204-cf54cec014f3" (UID: "88f88e61-2539-422d-b204-cf54cec014f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:03.995755 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.995735 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-credential-socket" pod "88f88e61-2539-422d-b204-cf54cec014f3" (UID: "88f88e61-2539-422d-b204-cf54cec014f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:03.996263 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.996238 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f88e61-2539-422d-b204-cf54cec014f3-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:03.996375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.996267 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:03.996375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.996282 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:03.996375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.996296 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:03.996375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.996309 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:03.997291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.997266 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88f88e61-2539-422d-b204-cf54cec014f3-istio-podinfo" pod "88f88e61-2539-422d-b204-cf54cec014f3" (UID: "88f88e61-2539-422d-b204-cf54cec014f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:03.997598 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.997578 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f88e61-2539-422d-b204-cf54cec014f3-kube-api-access-nwcjp" pod "88f88e61-2539-422d-b204-cf54cec014f3" (UID: "88f88e61-2539-422d-b204-cf54cec014f3"). InnerVolumeSpecName "kube-api-access-nwcjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:03.997738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.997608 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f88e61-2539-422d-b204-cf54cec014f3-istio-token" pod "88f88e61-2539-422d-b204-cf54cec014f3" (UID: "88f88e61-2539-422d-b204-cf54cec014f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:03.997876 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:03.997859 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-istio-envoy" pod "88f88e61-2539-422d-b204-cf54cec014f3" (UID: "88f88e61-2539-422d-b204-cf54cec014f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:04.069660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.069587 4335 generic.go:356] "Generic (PLEG): container finished" podID="769e08d0-bf5e-404e-8b7a-adf43dcd0905" containerID="18ba68e9205228fe0bf3ac3622860c1cd4bbd08d20f61b64ff632720e97009e1" exitCode=0 Oct 30 22:17:04.069856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.069681 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" event={"ID":"769e08d0-bf5e-404e-8b7a-adf43dcd0905","Type":"ContainerDied","Data":"18ba68e9205228fe0bf3ac3622860c1cd4bbd08d20f61b64ff632720e97009e1"} Oct 30 22:17:04.069856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.069736 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" event={"ID":"769e08d0-bf5e-404e-8b7a-adf43dcd0905","Type":"ContainerDied","Data":"d49efbdbf9f32a662e25e90304cbac4e31895c779a879beff5ff29a8eab133d4"} Oct 30 22:17:04.069856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.069762 4335 scope.go:122] "RemoveContainer" containerID="18ba68e9205228fe0bf3ac3622860c1cd4bbd08d20f61b64ff632720e97009e1" Oct 30 22:17:04.069856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.069690 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m" Oct 30 22:17:04.072125 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.072103 4335 generic.go:356] "Generic (PLEG): container finished" podID="88f88e61-2539-422d-b204-cf54cec014f3" containerID="b4c977a14769edffa5b840b1b35a81ecdc750bbfc3c7c65deb1f28b97f134d66" exitCode=0 Oct 30 22:17:04.072259 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.072151 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" Oct 30 22:17:04.072259 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.072198 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" event={"ID":"88f88e61-2539-422d-b204-cf54cec014f3","Type":"ContainerDied","Data":"b4c977a14769edffa5b840b1b35a81ecdc750bbfc3c7c65deb1f28b97f134d66"} Oct 30 22:17:04.072382 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.072259 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2" event={"ID":"88f88e61-2539-422d-b204-cf54cec014f3","Type":"ContainerDied","Data":"846c918482bc08e6d8f7162a5edd85480b0fbbb43229ab1f01ba1190a7603acc"} Oct 30 22:17:04.084784 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.083939 4335 scope.go:122] "RemoveContainer" containerID="18ba68e9205228fe0bf3ac3622860c1cd4bbd08d20f61b64ff632720e97009e1" Oct 30 22:17:04.084784 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:17:04.084239 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ba68e9205228fe0bf3ac3622860c1cd4bbd08d20f61b64ff632720e97009e1\": container with ID starting with 18ba68e9205228fe0bf3ac3622860c1cd4bbd08d20f61b64ff632720e97009e1 not found: ID does not exist" containerID="18ba68e9205228fe0bf3ac3622860c1cd4bbd08d20f61b64ff632720e97009e1" Oct 30 22:17:04.084784 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.084264 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ba68e9205228fe0bf3ac3622860c1cd4bbd08d20f61b64ff632720e97009e1"} err="failed to get container status \"18ba68e9205228fe0bf3ac3622860c1cd4bbd08d20f61b64ff632720e97009e1\": rpc error: code = NotFound desc = could not find container \"18ba68e9205228fe0bf3ac3622860c1cd4bbd08d20f61b64ff632720e97009e1\": container with ID starting with 18ba68e9205228fe0bf3ac3622860c1cd4bbd08d20f61b64ff632720e97009e1 not found: ID does not exist" Oct 30 22:17:04.084784 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.084282 4335 scope.go:122] "RemoveContainer" containerID="b4c977a14769edffa5b840b1b35a81ecdc750bbfc3c7c65deb1f28b97f134d66" Oct 30 22:17:04.103193 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.103166 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769e08d0-bf5e-404e-8b7a-adf43dcd0905-kube-api-access-r2rm5\" (UniqueName: \"kubernetes.io/projected/769e08d0-bf5e-404e-8b7a-adf43dcd0905-kube-api-access-r2rm5\") pod \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " Oct 30 22:17:04.103375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.103230 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-workload-socket\") pod \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " Oct 30 22:17:04.103375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.103260 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-podinfo\") pod \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " Oct 30 22:17:04.103375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.103291 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-envoy\") pod \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " Oct 30 22:17:04.103375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.103361 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-workload-certs\") pod \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " Oct 30 22:17:04.103598 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.103476 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-credential-socket\") pod \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " Oct 30 22:17:04.103598 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.103554 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istiod-ca-cert\") pod \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " Oct 30 22:17:04.103731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.103646 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-data\") pod \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " Oct 30 22:17:04.103731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.103705 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-token\" (UniqueName: \"kubernetes.io/projected/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-token\") pod \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\" (UID: \"769e08d0-bf5e-404e-8b7a-adf43dcd0905\") " Oct 30 22:17:04.104287 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.104267 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f88e61-2539-422d-b204-cf54cec014f3-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:04.104383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.104291 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f88e61-2539-422d-b204-cf54cec014f3-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:04.104383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.104305 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwcjp\" (UniqueName: \"kubernetes.io/projected/88f88e61-2539-422d-b204-cf54cec014f3-kube-api-access-nwcjp\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:04.104383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.104321 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f88e61-2539-422d-b204-cf54cec014f3-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:04.106578 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.106541 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-workload-socket" pod "769e08d0-bf5e-404e-8b7a-adf43dcd0905" (UID: "769e08d0-bf5e-404e-8b7a-adf43dcd0905"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:04.107154 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.107131 4335 scope.go:122] "RemoveContainer" containerID="b4c977a14769edffa5b840b1b35a81ecdc750bbfc3c7c65deb1f28b97f134d66" Oct 30 22:17:04.107834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.107408 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-credential-socket" pod "769e08d0-bf5e-404e-8b7a-adf43dcd0905" (UID: "769e08d0-bf5e-404e-8b7a-adf43dcd0905"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:04.107834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.107719 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istiod-ca-cert" pod "769e08d0-bf5e-404e-8b7a-adf43dcd0905" (UID: "769e08d0-bf5e-404e-8b7a-adf43dcd0905"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:04.108173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.108065 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-workload-certs" pod "769e08d0-bf5e-404e-8b7a-adf43dcd0905" (UID: "769e08d0-bf5e-404e-8b7a-adf43dcd0905"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:04.108274 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:17:04.108202 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c977a14769edffa5b840b1b35a81ecdc750bbfc3c7c65deb1f28b97f134d66\": container with ID starting with b4c977a14769edffa5b840b1b35a81ecdc750bbfc3c7c65deb1f28b97f134d66 not found: ID does not exist" containerID="b4c977a14769edffa5b840b1b35a81ecdc750bbfc3c7c65deb1f28b97f134d66" Oct 30 22:17:04.108274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.108237 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c977a14769edffa5b840b1b35a81ecdc750bbfc3c7c65deb1f28b97f134d66"} err="failed to get container status \"b4c977a14769edffa5b840b1b35a81ecdc750bbfc3c7c65deb1f28b97f134d66\": rpc error: code = NotFound desc = could not find container \"b4c977a14769edffa5b840b1b35a81ecdc750bbfc3c7c65deb1f28b97f134d66\": container with ID starting with b4c977a14769edffa5b840b1b35a81ecdc750bbfc3c7c65deb1f28b97f134d66 not found: ID does not exist" Oct 30 22:17:04.108433 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.108412 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-data" pod "769e08d0-bf5e-404e-8b7a-adf43dcd0905" (UID: "769e08d0-bf5e-404e-8b7a-adf43dcd0905"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:04.108847 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.108824 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-token" pod "769e08d0-bf5e-404e-8b7a-adf43dcd0905" (UID: "769e08d0-bf5e-404e-8b7a-adf43dcd0905"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:04.111051 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.111025 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769e08d0-bf5e-404e-8b7a-adf43dcd0905-kube-api-access-r2rm5" pod "769e08d0-bf5e-404e-8b7a-adf43dcd0905" (UID: "769e08d0-bf5e-404e-8b7a-adf43dcd0905"). InnerVolumeSpecName "kube-api-access-r2rm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:04.112020 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.111994 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-envoy" pod "769e08d0-bf5e-404e-8b7a-adf43dcd0905" (UID: "769e08d0-bf5e-404e-8b7a-adf43dcd0905"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:04.112379 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.112357 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-podinfo" pod "769e08d0-bf5e-404e-8b7a-adf43dcd0905" (UID: "769e08d0-bf5e-404e-8b7a-adf43dcd0905"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:04.207055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.207004 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:04.207055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.207031 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:04.207055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.207040 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:04.207055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.207049 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:04.207055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.207059 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2rm5\" (UniqueName: \"kubernetes.io/projected/769e08d0-bf5e-404e-8b7a-adf43dcd0905-kube-api-access-r2rm5\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:04.207667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.207076 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:04.207667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.207090 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:04.207667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.207100 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:04.207667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.207108 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e08d0-bf5e-404e-8b7a-adf43dcd0905-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:04.351422 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.351387 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2"] Oct 30 22:17:04.355695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.355635 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-224x2"] Oct 30 22:17:04.497558 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.497524 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m"] Oct 30 22:17:04.501742 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:04.501717 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-j4v8m"] Oct 30 22:17:04.863856 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:17:04.863827 4335 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice/crio-dc52df387192676062840a9952066c80ef74d1b37d55f395b1bf2bfde1da0894\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice\": RecentStats: unable to find data in memory cache]" Oct 30 22:17:05.058802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:05.058772 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="769e08d0-bf5e-404e-8b7a-adf43dcd0905" path="/var/lib/kubelet/pods/769e08d0-bf5e-404e-8b7a-adf43dcd0905/volumes" Oct 30 22:17:05.059282 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:05.059265 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88f88e61-2539-422d-b204-cf54cec014f3" path="/var/lib/kubelet/pods/88f88e61-2539-422d-b204-cf54cec014f3/volumes" Oct 30 22:17:06.291628 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:17:06.291571 4335 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice/crio-dc52df387192676062840a9952066c80ef74d1b37d55f395b1bf2bfde1da0894\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice\": RecentStats: unable to find data in memory cache]" Oct 30 22:17:12.415931 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.415899 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p"] Oct 30 22:17:12.416816 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.416794 4335 cpu_manager.go:414] "containerMap: removing container" podUID="88f88e61-2539-422d-b204-cf54cec014f3" containerName="istio-proxy" Oct 30 22:17:12.416816 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.416817 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88f88e61-2539-422d-b204-cf54cec014f3" containerName="istio-proxy" Oct 30 22:17:12.417009 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.416848 4335 cpu_manager.go:414] "containerMap: removing container" podUID="769e08d0-bf5e-404e-8b7a-adf43dcd0905" containerName="istio-proxy" Oct 30 22:17:12.417009 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.416857 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769e08d0-bf5e-404e-8b7a-adf43dcd0905" containerName="istio-proxy" Oct 30 22:17:12.417009 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.416899 4335 cpu_manager.go:414] "containerMap: removing container" podUID="aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" containerName="istio-proxy" Oct 30 22:17:12.417009 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.416907 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8ee860-7c1a-43f1-8e21-634a5bc0dee1" containerName="istio-proxy" Oct 30 22:17:12.417201 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.417064 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="88f88e61-2539-422d-b204-cf54cec014f3" containerName="istio-proxy" Oct 30 22:17:12.417201 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.417085 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="769e08d0-bf5e-404e-8b7a-adf43dcd0905" containerName="istio-proxy" Oct 30 22:17:12.422535 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.422513 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.495036 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.494992 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ec058c-openshift-default-dockercfg-th7ch\"" Oct 30 22:17:12.505227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.505069 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-workload-socket\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.505526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.505501 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-workload-certs\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.505688 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.505672 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-podinfo\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.505824 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.505810 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-credential-socket\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.505926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.505912 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istiod-ca-cert\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.506076 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.506061 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-token\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.506203 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.506189 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhhsr\" (UniqueName: \"kubernetes.io/projected/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-kube-api-access-jhhsr\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.506360 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.506345 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-envoy\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.506484 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.506471 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-data\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.594730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.594697 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p"] Oct 30 22:17:12.621956 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.621910 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-workload-certs\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.622161 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.621996 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-podinfo\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.622161 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.622042 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-credential-socket\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.622161 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.622076 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istiod-ca-cert\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.622161 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.622156 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-token\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.622398 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.622204 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhhsr\" (UniqueName: \"kubernetes.io/projected/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-kube-api-access-jhhsr\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.622398 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.622282 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-envoy\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.622398 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.622339 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-data\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.622564 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.622463 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-workload-socket\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.623005 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.622982 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-workload-socket\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.623257 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.623235 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-workload-certs\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.624257 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.624231 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-credential-socket\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.625024 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.624996 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istiod-ca-cert\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.627550 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.627501 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-podinfo\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.627805 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.627783 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-data\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.629183 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.629158 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-envoy\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.698636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.698540 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-token\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.701564 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.701534 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhhsr\" (UniqueName: \"kubernetes.io/projected/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-kube-api-access-jhhsr\") pod \"e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:12.751915 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:12.751888 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:13.246934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:13.246903 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p"] Oct 30 22:17:13.247359 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:17:13.247338 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f63a86_2ce4_45ad_8f6a_38025710f1f5.slice/crio-142b9de95e0ad30b461917333fcc4e1f66856845cb77ba2d0ce6aea7fc862418 WatchSource:0}: Error finding container 142b9de95e0ad30b461917333fcc4e1f66856845cb77ba2d0ce6aea7fc862418: Status 404 returned error can't find the container with id 142b9de95e0ad30b461917333fcc4e1f66856845cb77ba2d0ce6aea7fc862418 Oct 30 22:17:13.249939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:13.249907 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:17:13.250050 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:13.249978 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:17:13.250050 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:13.250023 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:17:14.160904 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:14.160871 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" event={"ID":"e7f63a86-2ce4-45ad-8f6a-38025710f1f5","Type":"ContainerStarted","Data":"797a88611d5a92c6fdd8203ced121e2795c14e6c30a5a18512daec383c220d05"} Oct 30 22:17:14.161368 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:14.160930 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" event={"ID":"e7f63a86-2ce4-45ad-8f6a-38025710f1f5","Type":"ContainerStarted","Data":"142b9de95e0ad30b461917333fcc4e1f66856845cb77ba2d0ce6aea7fc862418"} Oct 30 22:17:14.378953 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:14.378907 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" podStartSLOduration=2.378895891 podStartE2EDuration="2.378895891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:14.329481734 +0000 UTC m=+1967.429913469" watchObservedRunningTime="2026-10-30 22:17:14.378895891 +0000 UTC m=+1967.479327622" Oct 30 22:17:14.752137 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:14.752107 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:14.758859 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:14.758830 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:15.168864 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:15.168833 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:15.169788 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:15.169766 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:17:16.380366 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:17:16.380329 4335 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice/crio-dc52df387192676062840a9952066c80ef74d1b37d55f395b1bf2bfde1da0894\": RecentStats: unable to find data in memory cache]" Oct 30 22:17:19.788038 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:17:19.788000 4335 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice/crio-dc52df387192676062840a9952066c80ef74d1b37d55f395b1bf2bfde1da0894\": RecentStats: unable to find data in memory cache]" Oct 30 22:17:24.463770 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:17:24.463727 4335 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice/crio-dc52df387192676062840a9952066c80ef74d1b37d55f395b1bf2bfde1da0894\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice\": RecentStats: unable to find data in memory cache]" Oct 30 22:17:24.464192 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:17:24.463729 4335 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice/crio-dc52df387192676062840a9952066c80ef74d1b37d55f395b1bf2bfde1da0894\": RecentStats: unable to find data in memory cache]" Oct 30 22:17:25.326239 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:17:25.326200 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:17:26.395051 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:17:26.395017 4335 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice/crio-dc52df387192676062840a9952066c80ef74d1b37d55f395b1bf2bfde1da0894\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b2369a_d31f_4413_b5c6_ec4b217c3f8b.slice\": RecentStats: unable to find data in memory cache]" Oct 30 22:17:28.843675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:28.843238 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv"] Oct 30 22:17:29.019336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.019276 4335 scope.go:122] "RemoveContainer" containerID="29964145add9f9935c968107233c250f88ab7d502ac947a71081e6a8554db70d" Oct 30 22:17:29.037130 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.037089 4335 scope.go:122] "RemoveContainer" containerID="8b18918dadca3c336c53ffe48251244f7169eda446db1a666c2a20cb0e551b07" Oct 30 22:17:29.037379 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.037357 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv"] Oct 30 22:17:29.037464 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.037406 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb"] Oct 30 22:17:29.039291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.039267 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv" Oct 30 22:17:29.048199 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.048175 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb" Oct 30 22:17:29.055811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.055787 4335 scope.go:122] "RemoveContainer" containerID="77d19e00002b10042e3039793ae59e2a93e8a6778fdb68b4a7cbfb2a07e8d254" Oct 30 22:17:29.064409 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.064382 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c18511f-79df-44be-807e-f8d2744c7949-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv\" (UID: \"5c18511f-79df-44be-807e-f8d2744c7949\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv" Oct 30 22:17:29.064523 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.064508 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sk67\" (UniqueName: \"kubernetes.io/projected/5c18511f-79df-44be-807e-f8d2744c7949-kube-api-access-7sk67\") pod \"payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv\" (UID: \"5c18511f-79df-44be-807e-f8d2744c7949\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv" Oct 30 22:17:29.064627 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.064593 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfccb62-83d1-40d8-b0e4-8e04c9319490-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb\" (UID: \"fcfccb62-83d1-40d8-b0e4-8e04c9319490\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb" Oct 30 22:17:29.064834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.064814 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvm4w\" (UniqueName: \"kubernetes.io/projected/fcfccb62-83d1-40d8-b0e4-8e04c9319490-kube-api-access-nvm4w\") pod \"payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb\" (UID: \"fcfccb62-83d1-40d8-b0e4-8e04c9319490\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb" Oct 30 22:17:29.068308 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.068295 4335 scope.go:122] "RemoveContainer" containerID="2115741a634c367e939ba61aaf910c447d27efcd8b91b3ff67611948ee1ba263" Oct 30 22:17:29.079059 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.079040 4335 scope.go:122] "RemoveContainer" containerID="7a601301d43106135bb2a635c7596d0b32163ca9777490b9a8f7f992bd79c78a" Oct 30 22:17:29.081375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.081355 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb"] Oct 30 22:17:29.125681 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.124320 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-ec058c-dockercfg-qqf8v\"" Oct 30 22:17:29.125681 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.124363 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-ec058c\"" Oct 30 22:17:29.165947 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.165920 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvm4w\" (UniqueName: \"kubernetes.io/projected/fcfccb62-83d1-40d8-b0e4-8e04c9319490-kube-api-access-nvm4w\") pod \"payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb\" (UID: \"fcfccb62-83d1-40d8-b0e4-8e04c9319490\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb" Oct 30 22:17:29.166052 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.166012 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c18511f-79df-44be-807e-f8d2744c7949-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv\" (UID: \"5c18511f-79df-44be-807e-f8d2744c7949\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv" Oct 30 22:17:29.166109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.166095 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sk67\" (UniqueName: \"kubernetes.io/projected/5c18511f-79df-44be-807e-f8d2744c7949-kube-api-access-7sk67\") pod \"payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv\" (UID: \"5c18511f-79df-44be-807e-f8d2744c7949\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv" Oct 30 22:17:29.166158 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.166120 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfccb62-83d1-40d8-b0e4-8e04c9319490-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb\" (UID: \"fcfccb62-83d1-40d8-b0e4-8e04c9319490\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb" Oct 30 22:17:29.166881 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.166850 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c18511f-79df-44be-807e-f8d2744c7949-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv\" (UID: \"5c18511f-79df-44be-807e-f8d2744c7949\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv" Oct 30 22:17:29.167404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.167378 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfccb62-83d1-40d8-b0e4-8e04c9319490-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb\" (UID: \"fcfccb62-83d1-40d8-b0e4-8e04c9319490\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb" Oct 30 22:17:29.229064 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.229027 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvm4w\" (UniqueName: \"kubernetes.io/projected/fcfccb62-83d1-40d8-b0e4-8e04c9319490-kube-api-access-nvm4w\") pod \"payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb\" (UID: \"fcfccb62-83d1-40d8-b0e4-8e04c9319490\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb" Oct 30 22:17:29.229175 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.229113 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sk67\" (UniqueName: \"kubernetes.io/projected/5c18511f-79df-44be-807e-f8d2744c7949-kube-api-access-7sk67\") pod \"payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv\" (UID: \"5c18511f-79df-44be-807e-f8d2744c7949\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv" Oct 30 22:17:29.364434 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.364351 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv" Oct 30 22:17:29.375089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.375067 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb" Oct 30 22:17:29.764498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.764414 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv"] Oct 30 22:17:29.798470 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:17:29.798435 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c18511f_79df_44be_807e_f8d2744c7949.slice/crio-7ac287e7515c22d333e717cfe4ce6011f0b2bdd8624c405ad21a1a29f35ffb86 WatchSource:0}: Error finding container 7ac287e7515c22d333e717cfe4ce6011f0b2bdd8624c405ad21a1a29f35ffb86: Status 404 returned error can't find the container with id 7ac287e7515c22d333e717cfe4ce6011f0b2bdd8624c405ad21a1a29f35ffb86 Oct 30 22:17:29.815950 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:29.815922 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb"] Oct 30 22:17:29.816185 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:17:29.816161 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcfccb62_83d1_40d8_b0e4_8e04c9319490.slice/crio-7c36fb020de579307ea83de5676859cae962c31423116ba62154a717c3b66dbf WatchSource:0}: Error finding container 7c36fb020de579307ea83de5676859cae962c31423116ba62154a717c3b66dbf: Status 404 returned error can't find the container with id 7c36fb020de579307ea83de5676859cae962c31423116ba62154a717c3b66dbf Oct 30 22:17:30.309533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:30.309488 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv" event={"ID":"5c18511f-79df-44be-807e-f8d2744c7949","Type":"ContainerStarted","Data":"4886f68da238f2d21e3a9902c3f508592c4c273265378beee5b971c6bb54a97c"} Oct 30 22:17:30.310415 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:30.309563 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv" event={"ID":"5c18511f-79df-44be-807e-f8d2744c7949","Type":"ContainerStarted","Data":"7ac287e7515c22d333e717cfe4ce6011f0b2bdd8624c405ad21a1a29f35ffb86"} Oct 30 22:17:30.310415 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:30.309833 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv" Oct 30 22:17:30.311598 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:30.311569 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb" event={"ID":"fcfccb62-83d1-40d8-b0e4-8e04c9319490","Type":"ContainerStarted","Data":"1f9df4088b1d2dc85a51e60a4a26aff77b23fa4b84891fb1bb5be2ed856637ba"} Oct 30 22:17:30.311733 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:30.311665 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb" event={"ID":"fcfccb62-83d1-40d8-b0e4-8e04c9319490","Type":"ContainerStarted","Data":"7c36fb020de579307ea83de5676859cae962c31423116ba62154a717c3b66dbf"} Oct 30 22:17:30.311733 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:30.311688 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb" Oct 30 22:17:30.424701 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:30.424637 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv" podStartSLOduration=2.424600138 podStartE2EDuration="2.424600138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:30.424193895 +0000 UTC m=+1983.524625628" watchObservedRunningTime="2026-10-30 22:17:30.424600138 +0000 UTC m=+1983.525031871" Oct 30 22:17:30.544882 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:30.543767 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb" podStartSLOduration=2.543742635 podStartE2EDuration="2.543742635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:30.5421609 +0000 UTC m=+1983.642592650" watchObservedRunningTime="2026-10-30 22:17:30.543742635 +0000 UTC m=+1983.644174457" Oct 30 22:17:38.567083 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.567042 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn"] Oct 30 22:17:38.572002 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.571979 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.634684 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.626518 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a1024a-openshift-default-dockercfg-h76tn\"" Oct 30 22:17:38.682977 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.682929 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn"] Oct 30 22:17:38.684807 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.684776 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-workload-socket\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.684964 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.684874 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-workload-certs\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.684964 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.684906 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istiod-ca-cert\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.684964 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.684940 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-envoy\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.684964 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.684962 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-data\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.685223 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.685068 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-podinfo\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.685223 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.685106 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-token\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.685223 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.685176 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-credential-socket\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.685363 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.685272 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7ldl\" (UniqueName: \"kubernetes.io/projected/5e3c2933-2586-4001-aa99-c18c1a6d44bb-kube-api-access-b7ldl\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.787110 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.787075 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-credential-socket\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.787342 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.787187 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7ldl\" (UniqueName: \"kubernetes.io/projected/5e3c2933-2586-4001-aa99-c18c1a6d44bb-kube-api-access-b7ldl\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.787530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.787507 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-credential-socket\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.787690 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.787670 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-workload-socket\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.787780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.787767 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-workload-certs\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.787842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.787797 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istiod-ca-cert\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.787842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.787832 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-envoy\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.787943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.787856 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-data\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.787997 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.787974 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-podinfo\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.788054 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.788015 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-token\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.788109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.788058 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-workload-socket\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.788494 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.788469 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istiod-ca-cert\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.788783 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.788763 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-workload-certs\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.789065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.789042 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-data\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.791848 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.791792 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-envoy\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.792499 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.792469 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-podinfo\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.849377 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.849224 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7ldl\" (UniqueName: \"kubernetes.io/projected/5e3c2933-2586-4001-aa99-c18c1a6d44bb-kube-api-access-b7ldl\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.850872 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.850845 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-token\") pod \"e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:38.887474 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:38.887445 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:39.349552 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:17:39.349525 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3c2933_2586_4001_aa99_c18c1a6d44bb.slice/crio-c8b77cc4664dd7a03ca4b4dd52b2246ee3fb24f5d40f5ab7a106359e6a11cd15 WatchSource:0}: Error finding container c8b77cc4664dd7a03ca4b4dd52b2246ee3fb24f5d40f5ab7a106359e6a11cd15: Status 404 returned error can't find the container with id c8b77cc4664dd7a03ca4b4dd52b2246ee3fb24f5d40f5ab7a106359e6a11cd15 Oct 30 22:17:39.349790 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:39.349771 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn"] Oct 30 22:17:39.352118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:39.352085 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:17:39.352226 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:39.352146 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:17:39.352226 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:39.352175 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:17:39.385920 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:39.385888 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" event={"ID":"5e3c2933-2586-4001-aa99-c18c1a6d44bb","Type":"ContainerStarted","Data":"c8b77cc4664dd7a03ca4b4dd52b2246ee3fb24f5d40f5ab7a106359e6a11cd15"} Oct 30 22:17:40.396360 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:40.396324 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" event={"ID":"5e3c2933-2586-4001-aa99-c18c1a6d44bb","Type":"ContainerStarted","Data":"868c00326a18b5e45b5760dde9dfd0fb89c3bf7d9aff85cecde3eb47a5b3350a"} Oct 30 22:17:40.566060 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:40.566005 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" podStartSLOduration=2.565988993 podStartE2EDuration="2.565988993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:40.564469866 +0000 UTC m=+1993.664901599" watchObservedRunningTime="2026-10-30 22:17:40.565988993 +0000 UTC m=+1993.666420726" Oct 30 22:17:40.887959 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:40.887928 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:40.893967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:40.893942 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:41.322849 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:41.322821 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb" Oct 30 22:17:41.323041 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:41.322885 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv" Oct 30 22:17:41.405696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:41.405670 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:41.406743 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:41.406721 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:17:48.398144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:48.398105 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d"] Oct 30 22:17:48.402829 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:48.402812 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d" Oct 30 22:17:48.462813 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:48.462781 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d"] Oct 30 22:17:48.504142 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:48.504109 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:17:48.504495 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:48.504466 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r2sj\" (UniqueName: \"kubernetes.io/projected/0002b46b-ce78-46cd-847a-6b9c334715fd-kube-api-access-6r2sj\") pod \"maas-api-revoke-sub-9d8626ba-rlf9d\" (UID: \"0002b46b-ce78-46cd-847a-6b9c334715fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d" Oct 30 22:17:48.504714 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:48.504689 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0002b46b-ce78-46cd-847a-6b9c334715fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d8626ba-rlf9d\" (UID: \"0002b46b-ce78-46cd-847a-6b9c334715fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d" Oct 30 22:17:48.504799 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:48.504732 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0002b46b-ce78-46cd-847a-6b9c334715fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d8626ba-rlf9d\" (UID: \"0002b46b-ce78-46cd-847a-6b9c334715fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d" Oct 30 22:17:48.504876 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:48.504821 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:17:48.504934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:48.504892 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:17:48.504993 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:48.504939 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:17:48.606034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:48.606008 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r2sj\" (UniqueName: \"kubernetes.io/projected/0002b46b-ce78-46cd-847a-6b9c334715fd-kube-api-access-6r2sj\") pod \"maas-api-revoke-sub-9d8626ba-rlf9d\" (UID: \"0002b46b-ce78-46cd-847a-6b9c334715fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d" Oct 30 22:17:48.606190 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:48.606116 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0002b46b-ce78-46cd-847a-6b9c334715fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d8626ba-rlf9d\" (UID: \"0002b46b-ce78-46cd-847a-6b9c334715fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d" Oct 30 22:17:48.606190 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:48.606135 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0002b46b-ce78-46cd-847a-6b9c334715fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d8626ba-rlf9d\" (UID: \"0002b46b-ce78-46cd-847a-6b9c334715fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d" Oct 30 22:17:48.607029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:48.606986 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0002b46b-ce78-46cd-847a-6b9c334715fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d8626ba-rlf9d\" (UID: \"0002b46b-ce78-46cd-847a-6b9c334715fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d" Oct 30 22:17:48.608777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:48.608750 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0002b46b-ce78-46cd-847a-6b9c334715fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d8626ba-rlf9d\" (UID: \"0002b46b-ce78-46cd-847a-6b9c334715fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d" Oct 30 22:17:48.688220 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:48.688160 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r2sj\" (UniqueName: \"kubernetes.io/projected/0002b46b-ce78-46cd-847a-6b9c334715fd-kube-api-access-6r2sj\") pod \"maas-api-revoke-sub-9d8626ba-rlf9d\" (UID: \"0002b46b-ce78-46cd-847a-6b9c334715fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d" Oct 30 22:17:48.718685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:48.718473 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d" Oct 30 22:17:49.122396 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:17:49.122363 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0002b46b_ce78_46cd_847a_6b9c334715fd.slice/crio-a09b0b6773c357349bacab87a4637eaff2edf3b84d3e262b6ba647e667840daa WatchSource:0}: Error finding container a09b0b6773c357349bacab87a4637eaff2edf3b84d3e262b6ba647e667840daa: Status 404 returned error can't find the container with id a09b0b6773c357349bacab87a4637eaff2edf3b84d3e262b6ba647e667840daa Oct 30 22:17:49.124568 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:49.124541 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d"] Oct 30 22:17:49.523431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:49.523390 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d" event={"ID":"0002b46b-ce78-46cd-847a-6b9c334715fd","Type":"ContainerStarted","Data":"62239c8f2f3bb8bc09eb40e08e5ed673e087a1ee23c701e789a90165971a48fe"} Oct 30 22:17:49.523945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:49.523453 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d" event={"ID":"0002b46b-ce78-46cd-847a-6b9c334715fd","Type":"ContainerStarted","Data":"a09b0b6773c357349bacab87a4637eaff2edf3b84d3e262b6ba647e667840daa"} Oct 30 22:17:49.644626 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:49.644563 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d" podStartSLOduration=1.6445489690000001 podStartE2EDuration="1.644548969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:49.640259931 +0000 UTC m=+2002.740691663" watchObservedRunningTime="2026-10-30 22:17:49.644548969 +0000 UTC m=+2002.744980701" Oct 30 22:17:50.537910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:50.537815 4335 generic.go:356] "Generic (PLEG): container finished" podID="0002b46b-ce78-46cd-847a-6b9c334715fd" containerID="62239c8f2f3bb8bc09eb40e08e5ed673e087a1ee23c701e789a90165971a48fe" exitCode=0 Oct 30 22:17:50.538367 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:50.537904 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d" event={"ID":"0002b46b-ce78-46cd-847a-6b9c334715fd","Type":"ContainerDied","Data":"62239c8f2f3bb8bc09eb40e08e5ed673e087a1ee23c701e789a90165971a48fe"} Oct 30 22:17:51.717497 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:51.717474 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d" Oct 30 22:17:51.862427 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:51.862395 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0002b46b-ce78-46cd-847a-6b9c334715fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0002b46b-ce78-46cd-847a-6b9c334715fd-maas-api-openshift-ca\") pod \"0002b46b-ce78-46cd-847a-6b9c334715fd\" (UID: \"0002b46b-ce78-46cd-847a-6b9c334715fd\") " Oct 30 22:17:51.862609 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:51.862485 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0002b46b-ce78-46cd-847a-6b9c334715fd-kube-api-access-6r2sj\" (UniqueName: \"kubernetes.io/projected/0002b46b-ce78-46cd-847a-6b9c334715fd-kube-api-access-6r2sj\") pod \"0002b46b-ce78-46cd-847a-6b9c334715fd\" (UID: \"0002b46b-ce78-46cd-847a-6b9c334715fd\") " Oct 30 22:17:51.862609 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:51.862564 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0002b46b-ce78-46cd-847a-6b9c334715fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0002b46b-ce78-46cd-847a-6b9c334715fd-maas-api-xks-ca\") pod \"0002b46b-ce78-46cd-847a-6b9c334715fd\" (UID: \"0002b46b-ce78-46cd-847a-6b9c334715fd\") " Oct 30 22:17:51.862936 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:51.862904 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0002b46b-ce78-46cd-847a-6b9c334715fd-maas-api-openshift-ca" pod "0002b46b-ce78-46cd-847a-6b9c334715fd" (UID: "0002b46b-ce78-46cd-847a-6b9c334715fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:51.863370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:51.863268 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0002b46b-ce78-46cd-847a-6b9c334715fd-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:51.864921 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:51.864894 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0002b46b-ce78-46cd-847a-6b9c334715fd-kube-api-access-6r2sj" pod "0002b46b-ce78-46cd-847a-6b9c334715fd" (UID: "0002b46b-ce78-46cd-847a-6b9c334715fd"). InnerVolumeSpecName "kube-api-access-6r2sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:51.865352 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:51.865319 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0002b46b-ce78-46cd-847a-6b9c334715fd-maas-api-xks-ca" pod "0002b46b-ce78-46cd-847a-6b9c334715fd" (UID: "0002b46b-ce78-46cd-847a-6b9c334715fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:51.964526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:51.964500 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r2sj\" (UniqueName: \"kubernetes.io/projected/0002b46b-ce78-46cd-847a-6b9c334715fd-kube-api-access-6r2sj\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:51.964526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:51.964530 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0002b46b-ce78-46cd-847a-6b9c334715fd-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:52.052288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:52.052263 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97"] Oct 30 22:17:52.056551 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:52.056530 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a997efcc-n7g97"] Oct 30 22:17:52.557517 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:52.557491 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d" Oct 30 22:17:52.557712 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:52.557536 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d" event={"ID":"0002b46b-ce78-46cd-847a-6b9c334715fd","Type":"ContainerDied","Data":"a09b0b6773c357349bacab87a4637eaff2edf3b84d3e262b6ba647e667840daa"} Oct 30 22:17:52.557712 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:52.557595 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a09b0b6773c357349bacab87a4637eaff2edf3b84d3e262b6ba647e667840daa" Oct 30 22:17:53.060289 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:17:53.060259 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e3f076a-bb85-4f0e-bcd9-ef5cc5101516" path="/var/lib/kubelet/pods/3e3f076a-bb85-4f0e-bcd9-ef5cc5101516/volumes" Oct 30 22:17:55.326956 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:17:55.326917 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:18:04.223662 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:04.223630 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:18:06.057891 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.057857 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx"] Oct 30 22:18:06.058597 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.058581 4335 cpu_manager.go:414] "containerMap: removing container" podUID="0002b46b-ce78-46cd-847a-6b9c334715fd" containerName="revoke-keys" Oct 30 22:18:06.058690 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.058599 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0002b46b-ce78-46cd-847a-6b9c334715fd" containerName="revoke-keys" Oct 30 22:18:06.058757 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.058747 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="0002b46b-ce78-46cd-847a-6b9c334715fd" containerName="revoke-keys" Oct 30 22:18:06.065547 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.065528 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx" Oct 30 22:18:06.118998 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.118973 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx"] Oct 30 22:18:06.162520 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.162491 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:18:06.162520 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.162499 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:18:06.162520 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.162513 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:18:06.162789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.162563 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:18:06.216871 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.216834 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm6tl\" (UniqueName: \"kubernetes.io/projected/f17319ef-ff41-4310-8126-4f50ec390d5f-kube-api-access-cm6tl\") pod \"maas-api-revoke-keys-e2e-mt-3b355441-ft2bx\" (UID: \"f17319ef-ff41-4310-8126-4f50ec390d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx" Oct 30 22:18:06.217016 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.217000 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f17319ef-ff41-4310-8126-4f50ec390d5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3b355441-ft2bx\" (UID: \"f17319ef-ff41-4310-8126-4f50ec390d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx" Oct 30 22:18:06.217207 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.217186 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f17319ef-ff41-4310-8126-4f50ec390d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3b355441-ft2bx\" (UID: \"f17319ef-ff41-4310-8126-4f50ec390d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx" Oct 30 22:18:06.318666 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.318557 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f17319ef-ff41-4310-8126-4f50ec390d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3b355441-ft2bx\" (UID: \"f17319ef-ff41-4310-8126-4f50ec390d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx" Oct 30 22:18:06.318820 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.318708 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm6tl\" (UniqueName: \"kubernetes.io/projected/f17319ef-ff41-4310-8126-4f50ec390d5f-kube-api-access-cm6tl\") pod \"maas-api-revoke-keys-e2e-mt-3b355441-ft2bx\" (UID: \"f17319ef-ff41-4310-8126-4f50ec390d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx" Oct 30 22:18:06.318820 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.318792 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f17319ef-ff41-4310-8126-4f50ec390d5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3b355441-ft2bx\" (UID: \"f17319ef-ff41-4310-8126-4f50ec390d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx" Oct 30 22:18:06.319206 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.319185 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f17319ef-ff41-4310-8126-4f50ec390d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3b355441-ft2bx\" (UID: \"f17319ef-ff41-4310-8126-4f50ec390d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx" Oct 30 22:18:06.321243 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.321223 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f17319ef-ff41-4310-8126-4f50ec390d5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3b355441-ft2bx\" (UID: \"f17319ef-ff41-4310-8126-4f50ec390d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx" Oct 30 22:18:06.379097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.379073 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm6tl\" (UniqueName: \"kubernetes.io/projected/f17319ef-ff41-4310-8126-4f50ec390d5f-kube-api-access-cm6tl\") pod \"maas-api-revoke-keys-e2e-mt-3b355441-ft2bx\" (UID: \"f17319ef-ff41-4310-8126-4f50ec390d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx" Oct 30 22:18:06.379236 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.379216 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx" Oct 30 22:18:06.766701 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:06.766675 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx"] Oct 30 22:18:06.768245 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:18:06.768218 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17319ef_ff41_4310_8126_4f50ec390d5f.slice/crio-a1ffdb93923200e409edda064b588b62b80686374f9ce133aad4178e06fa86d4 WatchSource:0}: Error finding container a1ffdb93923200e409edda064b588b62b80686374f9ce133aad4178e06fa86d4: Status 404 returned error can't find the container with id a1ffdb93923200e409edda064b588b62b80686374f9ce133aad4178e06fa86d4 Oct 30 22:18:07.710263 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:07.710225 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx" event={"ID":"f17319ef-ff41-4310-8126-4f50ec390d5f","Type":"ContainerStarted","Data":"a7acea7405f1a8cddec02b81afd86f638b49cf7954fd71fbfe7c113f063857e5"} Oct 30 22:18:07.710924 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:07.710286 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx" event={"ID":"f17319ef-ff41-4310-8126-4f50ec390d5f","Type":"ContainerStarted","Data":"a1ffdb93923200e409edda064b588b62b80686374f9ce133aad4178e06fa86d4"} Oct 30 22:18:07.829120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:07.829064 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx" podStartSLOduration=1.829051185 podStartE2EDuration="1.829051185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:07.824783466 +0000 UTC m=+2020.925215197" watchObservedRunningTime="2026-10-30 22:18:07.829051185 +0000 UTC m=+2020.929482918" Oct 30 22:18:08.768357 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:08.768250 4335 generic.go:356] "Generic (PLEG): container finished" podID="f17319ef-ff41-4310-8126-4f50ec390d5f" containerID="a7acea7405f1a8cddec02b81afd86f638b49cf7954fd71fbfe7c113f063857e5" exitCode=0 Oct 30 22:18:08.770881 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:08.768576 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx" event={"ID":"f17319ef-ff41-4310-8126-4f50ec390d5f","Type":"ContainerDied","Data":"a7acea7405f1a8cddec02b81afd86f638b49cf7954fd71fbfe7c113f063857e5"} Oct 30 22:18:09.975873 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:09.975847 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx" Oct 30 22:18:10.087365 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:10.087335 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f17319ef-ff41-4310-8126-4f50ec390d5f-kube-api-access-cm6tl\" (UniqueName: \"kubernetes.io/projected/f17319ef-ff41-4310-8126-4f50ec390d5f-kube-api-access-cm6tl\") pod \"f17319ef-ff41-4310-8126-4f50ec390d5f\" (UID: \"f17319ef-ff41-4310-8126-4f50ec390d5f\") " Oct 30 22:18:10.087672 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:10.087398 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f17319ef-ff41-4310-8126-4f50ec390d5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f17319ef-ff41-4310-8126-4f50ec390d5f-maas-api-openshift-ca\") pod \"f17319ef-ff41-4310-8126-4f50ec390d5f\" (UID: \"f17319ef-ff41-4310-8126-4f50ec390d5f\") " Oct 30 22:18:10.087672 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:10.087570 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f17319ef-ff41-4310-8126-4f50ec390d5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f17319ef-ff41-4310-8126-4f50ec390d5f-maas-api-xks-ca\") pod \"f17319ef-ff41-4310-8126-4f50ec390d5f\" (UID: \"f17319ef-ff41-4310-8126-4f50ec390d5f\") " Oct 30 22:18:10.087901 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:10.087765 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f17319ef-ff41-4310-8126-4f50ec390d5f-maas-api-openshift-ca" pod "f17319ef-ff41-4310-8126-4f50ec390d5f" (UID: "f17319ef-ff41-4310-8126-4f50ec390d5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:10.088145 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:10.088118 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f17319ef-ff41-4310-8126-4f50ec390d5f-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:10.089836 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:10.089811 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17319ef-ff41-4310-8126-4f50ec390d5f-kube-api-access-cm6tl" pod "f17319ef-ff41-4310-8126-4f50ec390d5f" (UID: "f17319ef-ff41-4310-8126-4f50ec390d5f"). InnerVolumeSpecName "kube-api-access-cm6tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:10.089970 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:10.089946 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f17319ef-ff41-4310-8126-4f50ec390d5f-maas-api-xks-ca" pod "f17319ef-ff41-4310-8126-4f50ec390d5f" (UID: "f17319ef-ff41-4310-8126-4f50ec390d5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:10.189995 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:10.189972 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f17319ef-ff41-4310-8126-4f50ec390d5f-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:10.190109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:10.190014 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm6tl\" (UniqueName: \"kubernetes.io/projected/f17319ef-ff41-4310-8126-4f50ec390d5f-kube-api-access-cm6tl\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:10.785922 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:10.785895 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx" Oct 30 22:18:10.786097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:10.785937 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx" event={"ID":"f17319ef-ff41-4310-8126-4f50ec390d5f","Type":"ContainerDied","Data":"a1ffdb93923200e409edda064b588b62b80686374f9ce133aad4178e06fa86d4"} Oct 30 22:18:10.786097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:10.785983 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1ffdb93923200e409edda064b588b62b80686374f9ce133aad4178e06fa86d4" Oct 30 22:18:12.612882 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.612846 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj"] Oct 30 22:18:12.613901 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.613880 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f17319ef-ff41-4310-8126-4f50ec390d5f" containerName="revoke-keys" Oct 30 22:18:12.613979 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.613903 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f17319ef-ff41-4310-8126-4f50ec390d5f" containerName="revoke-keys" Oct 30 22:18:12.614128 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.614117 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f17319ef-ff41-4310-8126-4f50ec390d5f" containerName="revoke-keys" Oct 30 22:18:12.618491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.618470 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj" Oct 30 22:18:12.670508 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.670481 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:18:12.673574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.673552 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj"] Oct 30 22:18:12.717281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.717258 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:18:12.717404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.717284 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:18:12.717470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.717305 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:18:12.728144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.728116 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1122f4c8-84b1-46f0-825e-a51e2fb85b73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2298aa0-vffcj\" (UID: \"1122f4c8-84b1-46f0-825e-a51e2fb85b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj" Oct 30 22:18:12.728284 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.728195 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1122f4c8-84b1-46f0-825e-a51e2fb85b73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2298aa0-vffcj\" (UID: \"1122f4c8-84b1-46f0-825e-a51e2fb85b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj" Oct 30 22:18:12.728471 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.728444 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhsc4\" (UniqueName: \"kubernetes.io/projected/1122f4c8-84b1-46f0-825e-a51e2fb85b73-kube-api-access-nhsc4\") pod \"maas-api-revoke-sub-d2298aa0-vffcj\" (UID: \"1122f4c8-84b1-46f0-825e-a51e2fb85b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj" Oct 30 22:18:12.830817 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.830789 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1122f4c8-84b1-46f0-825e-a51e2fb85b73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2298aa0-vffcj\" (UID: \"1122f4c8-84b1-46f0-825e-a51e2fb85b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj" Oct 30 22:18:12.830978 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.830960 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhsc4\" (UniqueName: \"kubernetes.io/projected/1122f4c8-84b1-46f0-825e-a51e2fb85b73-kube-api-access-nhsc4\") pod \"maas-api-revoke-sub-d2298aa0-vffcj\" (UID: \"1122f4c8-84b1-46f0-825e-a51e2fb85b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj" Oct 30 22:18:12.831097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.831082 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1122f4c8-84b1-46f0-825e-a51e2fb85b73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2298aa0-vffcj\" (UID: \"1122f4c8-84b1-46f0-825e-a51e2fb85b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj" Oct 30 22:18:12.832108 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.832081 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1122f4c8-84b1-46f0-825e-a51e2fb85b73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2298aa0-vffcj\" (UID: \"1122f4c8-84b1-46f0-825e-a51e2fb85b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj" Oct 30 22:18:12.833601 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.833570 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1122f4c8-84b1-46f0-825e-a51e2fb85b73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2298aa0-vffcj\" (UID: \"1122f4c8-84b1-46f0-825e-a51e2fb85b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj" Oct 30 22:18:12.891799 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.891734 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhsc4\" (UniqueName: \"kubernetes.io/projected/1122f4c8-84b1-46f0-825e-a51e2fb85b73-kube-api-access-nhsc4\") pod \"maas-api-revoke-sub-d2298aa0-vffcj\" (UID: \"1122f4c8-84b1-46f0-825e-a51e2fb85b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj" Oct 30 22:18:12.934754 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:12.934732 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj" Oct 30 22:18:13.338203 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:13.338166 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj"] Oct 30 22:18:13.344150 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:18:13.344121 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1122f4c8_84b1_46f0_825e_a51e2fb85b73.slice/crio-dca67c226ebb74c423bc0d969261173fae403462116134e47e88626befcf2007 WatchSource:0}: Error finding container dca67c226ebb74c423bc0d969261173fae403462116134e47e88626befcf2007: Status 404 returned error can't find the container with id dca67c226ebb74c423bc0d969261173fae403462116134e47e88626befcf2007 Oct 30 22:18:13.825944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:13.825909 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj" event={"ID":"1122f4c8-84b1-46f0-825e-a51e2fb85b73","Type":"ContainerStarted","Data":"b6776e8ea5053f4465247a881e8aa494745be2e51836188c0a9af5b7f84ca7ce"} Oct 30 22:18:13.826404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:13.825974 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj" event={"ID":"1122f4c8-84b1-46f0-825e-a51e2fb85b73","Type":"ContainerStarted","Data":"dca67c226ebb74c423bc0d969261173fae403462116134e47e88626befcf2007"} Oct 30 22:18:13.944146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:13.944086 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj" podStartSLOduration=1.944068207 podStartE2EDuration="1.944068207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:13.941362315 +0000 UTC m=+2027.041794048" watchObservedRunningTime="2026-10-30 22:18:13.944068207 +0000 UTC m=+2027.044499939" Oct 30 22:18:14.848854 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:14.848822 4335 generic.go:356] "Generic (PLEG): container finished" podID="1122f4c8-84b1-46f0-825e-a51e2fb85b73" containerID="b6776e8ea5053f4465247a881e8aa494745be2e51836188c0a9af5b7f84ca7ce" exitCode=0 Oct 30 22:18:14.849252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:14.848887 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj" event={"ID":"1122f4c8-84b1-46f0-825e-a51e2fb85b73","Type":"ContainerDied","Data":"b6776e8ea5053f4465247a881e8aa494745be2e51836188c0a9af5b7f84ca7ce"} Oct 30 22:18:16.023297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:16.023271 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj" Oct 30 22:18:16.078223 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:16.078198 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1122f4c8-84b1-46f0-825e-a51e2fb85b73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1122f4c8-84b1-46f0-825e-a51e2fb85b73-maas-api-openshift-ca\") pod \"1122f4c8-84b1-46f0-825e-a51e2fb85b73\" (UID: \"1122f4c8-84b1-46f0-825e-a51e2fb85b73\") " Oct 30 22:18:16.078369 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:16.078248 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1122f4c8-84b1-46f0-825e-a51e2fb85b73-kube-api-access-nhsc4\" (UniqueName: \"kubernetes.io/projected/1122f4c8-84b1-46f0-825e-a51e2fb85b73-kube-api-access-nhsc4\") pod \"1122f4c8-84b1-46f0-825e-a51e2fb85b73\" (UID: \"1122f4c8-84b1-46f0-825e-a51e2fb85b73\") " Oct 30 22:18:16.078437 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:16.078367 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1122f4c8-84b1-46f0-825e-a51e2fb85b73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1122f4c8-84b1-46f0-825e-a51e2fb85b73-maas-api-xks-ca\") pod \"1122f4c8-84b1-46f0-825e-a51e2fb85b73\" (UID: \"1122f4c8-84b1-46f0-825e-a51e2fb85b73\") " Oct 30 22:18:16.078529 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:16.078504 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1122f4c8-84b1-46f0-825e-a51e2fb85b73-maas-api-openshift-ca" pod "1122f4c8-84b1-46f0-825e-a51e2fb85b73" (UID: "1122f4c8-84b1-46f0-825e-a51e2fb85b73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:16.078915 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:16.078893 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1122f4c8-84b1-46f0-825e-a51e2fb85b73-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.080518 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:16.080496 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1122f4c8-84b1-46f0-825e-a51e2fb85b73-kube-api-access-nhsc4" pod "1122f4c8-84b1-46f0-825e-a51e2fb85b73" (UID: "1122f4c8-84b1-46f0-825e-a51e2fb85b73"). InnerVolumeSpecName "kube-api-access-nhsc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:16.080648 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:16.080531 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1122f4c8-84b1-46f0-825e-a51e2fb85b73-maas-api-xks-ca" pod "1122f4c8-84b1-46f0-825e-a51e2fb85b73" (UID: "1122f4c8-84b1-46f0-825e-a51e2fb85b73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:16.179856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:16.179787 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1122f4c8-84b1-46f0-825e-a51e2fb85b73-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.179856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:16.179813 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhsc4\" (UniqueName: \"kubernetes.io/projected/1122f4c8-84b1-46f0-825e-a51e2fb85b73-kube-api-access-nhsc4\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.455738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:16.455661 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m"] Oct 30 22:18:16.567531 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:16.567467 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks"] Oct 30 22:18:16.878638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:16.867598 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj" event={"ID":"1122f4c8-84b1-46f0-825e-a51e2fb85b73","Type":"ContainerDied","Data":"dca67c226ebb74c423bc0d969261173fae403462116134e47e88626befcf2007"} Oct 30 22:18:16.878638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:16.867686 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dca67c226ebb74c423bc0d969261173fae403462116134e47e88626befcf2007" Oct 30 22:18:16.878638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:16.867764 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj" Oct 30 22:18:21.456008 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.455942 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m" podUID="67957741-7440-4dd3-9ddc-10dc3963b5f3" containerName="payload-processing" containerID="cri-o://e5d8ff981f796bd84046c9ae276f5699fbe7be6de46a749c66bd6e6d1991f4df" gracePeriod=25 Oct 30 22:18:21.568725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.568595 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks" podUID="9a6d6940-a2e0-43fa-86af-0f84f6a22a39" containerName="payload-pre-processing" containerID="cri-o://599b9df090b1d30d0a9037a0662cd44a2624bc7b7a33248fbb6e2b700f9a3097" gracePeriod=25 Oct 30 22:18:21.769185 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.768816 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m" Oct 30 22:18:21.894481 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.894440 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67957741-7440-4dd3-9ddc-10dc3963b5f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67957741-7440-4dd3-9ddc-10dc3963b5f3-plugins-config-volume\") pod \"67957741-7440-4dd3-9ddc-10dc3963b5f3\" (UID: \"67957741-7440-4dd3-9ddc-10dc3963b5f3\") " Oct 30 22:18:21.894697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.894550 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67957741-7440-4dd3-9ddc-10dc3963b5f3-kube-api-access-bq7tq\" (UniqueName: \"kubernetes.io/projected/67957741-7440-4dd3-9ddc-10dc3963b5f3-kube-api-access-bq7tq\") pod \"67957741-7440-4dd3-9ddc-10dc3963b5f3\" (UID: \"67957741-7440-4dd3-9ddc-10dc3963b5f3\") " Oct 30 22:18:21.895847 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.895754 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67957741-7440-4dd3-9ddc-10dc3963b5f3-plugins-config-volume" pod "67957741-7440-4dd3-9ddc-10dc3963b5f3" (UID: "67957741-7440-4dd3-9ddc-10dc3963b5f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:21.896753 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.896725 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks" Oct 30 22:18:21.897916 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.897874 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67957741-7440-4dd3-9ddc-10dc3963b5f3-kube-api-access-bq7tq" pod "67957741-7440-4dd3-9ddc-10dc3963b5f3" (UID: "67957741-7440-4dd3-9ddc-10dc3963b5f3"). InnerVolumeSpecName "kube-api-access-bq7tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:21.919684 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.919636 4335 generic.go:356] "Generic (PLEG): container finished" podID="9a6d6940-a2e0-43fa-86af-0f84f6a22a39" containerID="599b9df090b1d30d0a9037a0662cd44a2624bc7b7a33248fbb6e2b700f9a3097" exitCode=0 Oct 30 22:18:21.920071 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.919775 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks" event={"ID":"9a6d6940-a2e0-43fa-86af-0f84f6a22a39","Type":"ContainerDied","Data":"599b9df090b1d30d0a9037a0662cd44a2624bc7b7a33248fbb6e2b700f9a3097"} Oct 30 22:18:21.920071 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.919866 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks" event={"ID":"9a6d6940-a2e0-43fa-86af-0f84f6a22a39","Type":"ContainerDied","Data":"d68bd0cff3332678942fb748904ab96698ba79d7f2ea17fb587a1b77ce58b8ac"} Oct 30 22:18:21.920071 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.919901 4335 scope.go:122] "RemoveContainer" containerID="599b9df090b1d30d0a9037a0662cd44a2624bc7b7a33248fbb6e2b700f9a3097" Oct 30 22:18:21.920333 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.920093 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks" Oct 30 22:18:21.923977 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.923937 4335 generic.go:356] "Generic (PLEG): container finished" podID="67957741-7440-4dd3-9ddc-10dc3963b5f3" containerID="e5d8ff981f796bd84046c9ae276f5699fbe7be6de46a749c66bd6e6d1991f4df" exitCode=0 Oct 30 22:18:21.924637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.924508 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m" event={"ID":"67957741-7440-4dd3-9ddc-10dc3963b5f3","Type":"ContainerDied","Data":"e5d8ff981f796bd84046c9ae276f5699fbe7be6de46a749c66bd6e6d1991f4df"} Oct 30 22:18:21.924637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.924577 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m" event={"ID":"67957741-7440-4dd3-9ddc-10dc3963b5f3","Type":"ContainerDied","Data":"bc2cf9af7058b7e712722adaec3a7f4ec2bddce856ea3178344f4e2756210850"} Oct 30 22:18:21.924956 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.924921 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m" Oct 30 22:18:21.947877 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.947826 4335 scope.go:122] "RemoveContainer" containerID="599b9df090b1d30d0a9037a0662cd44a2624bc7b7a33248fbb6e2b700f9a3097" Oct 30 22:18:21.953643 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:18:21.948289 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599b9df090b1d30d0a9037a0662cd44a2624bc7b7a33248fbb6e2b700f9a3097\": container with ID starting with 599b9df090b1d30d0a9037a0662cd44a2624bc7b7a33248fbb6e2b700f9a3097 not found: ID does not exist" containerID="599b9df090b1d30d0a9037a0662cd44a2624bc7b7a33248fbb6e2b700f9a3097" Oct 30 22:18:21.953643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.948386 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599b9df090b1d30d0a9037a0662cd44a2624bc7b7a33248fbb6e2b700f9a3097"} err="failed to get container status \"599b9df090b1d30d0a9037a0662cd44a2624bc7b7a33248fbb6e2b700f9a3097\": rpc error: code = NotFound desc = could not find container \"599b9df090b1d30d0a9037a0662cd44a2624bc7b7a33248fbb6e2b700f9a3097\": container with ID starting with 599b9df090b1d30d0a9037a0662cd44a2624bc7b7a33248fbb6e2b700f9a3097 not found: ID does not exist" Oct 30 22:18:21.953643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.948423 4335 scope.go:122] "RemoveContainer" containerID="e5d8ff981f796bd84046c9ae276f5699fbe7be6de46a749c66bd6e6d1991f4df" Oct 30 22:18:21.976138 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.976117 4335 scope.go:122] "RemoveContainer" containerID="e5d8ff981f796bd84046c9ae276f5699fbe7be6de46a749c66bd6e6d1991f4df" Oct 30 22:18:21.976548 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:18:21.976524 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d8ff981f796bd84046c9ae276f5699fbe7be6de46a749c66bd6e6d1991f4df\": container with ID starting with e5d8ff981f796bd84046c9ae276f5699fbe7be6de46a749c66bd6e6d1991f4df not found: ID does not exist" containerID="e5d8ff981f796bd84046c9ae276f5699fbe7be6de46a749c66bd6e6d1991f4df" Oct 30 22:18:21.976686 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.976557 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d8ff981f796bd84046c9ae276f5699fbe7be6de46a749c66bd6e6d1991f4df"} err="failed to get container status \"e5d8ff981f796bd84046c9ae276f5699fbe7be6de46a749c66bd6e6d1991f4df\": rpc error: code = NotFound desc = could not find container \"e5d8ff981f796bd84046c9ae276f5699fbe7be6de46a749c66bd6e6d1991f4df\": container with ID starting with e5d8ff981f796bd84046c9ae276f5699fbe7be6de46a749c66bd6e6d1991f4df not found: ID does not exist" Oct 30 22:18:21.995863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.995840 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6d6940-a2e0-43fa-86af-0f84f6a22a39-kube-api-access-fmq6x\" (UniqueName: \"kubernetes.io/projected/9a6d6940-a2e0-43fa-86af-0f84f6a22a39-kube-api-access-fmq6x\") pod \"9a6d6940-a2e0-43fa-86af-0f84f6a22a39\" (UID: \"9a6d6940-a2e0-43fa-86af-0f84f6a22a39\") " Oct 30 22:18:21.996004 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.995991 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a6d6940-a2e0-43fa-86af-0f84f6a22a39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6d6940-a2e0-43fa-86af-0f84f6a22a39-plugins-config-volume\") pod \"9a6d6940-a2e0-43fa-86af-0f84f6a22a39\" (UID: \"9a6d6940-a2e0-43fa-86af-0f84f6a22a39\") " Oct 30 22:18:21.996370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.996348 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6d6940-a2e0-43fa-86af-0f84f6a22a39-plugins-config-volume" pod "9a6d6940-a2e0-43fa-86af-0f84f6a22a39" (UID: "9a6d6940-a2e0-43fa-86af-0f84f6a22a39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:21.996645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.996377 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq7tq\" (UniqueName: \"kubernetes.io/projected/67957741-7440-4dd3-9ddc-10dc3963b5f3-kube-api-access-bq7tq\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:21.996645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.996394 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67957741-7440-4dd3-9ddc-10dc3963b5f3-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:21.998033 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:21.998013 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6d6940-a2e0-43fa-86af-0f84f6a22a39-kube-api-access-fmq6x" pod "9a6d6940-a2e0-43fa-86af-0f84f6a22a39" (UID: "9a6d6940-a2e0-43fa-86af-0f84f6a22a39"). InnerVolumeSpecName "kube-api-access-fmq6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:22.098475 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:22.098443 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmq6x\" (UniqueName: \"kubernetes.io/projected/9a6d6940-a2e0-43fa-86af-0f84f6a22a39-kube-api-access-fmq6x\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:22.098475 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:22.098479 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6d6940-a2e0-43fa-86af-0f84f6a22a39-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:22.179950 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:22.179881 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m"] Oct 30 22:18:22.186109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:22.186084 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3b355441-57446bc65-j5t4m"] Oct 30 22:18:22.345951 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:22.345916 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks"] Oct 30 22:18:22.354954 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:22.354929 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3b355441-75c768b7ff-spjks"] Oct 30 22:18:23.059020 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:23.058992 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67957741-7440-4dd3-9ddc-10dc3963b5f3" path="/var/lib/kubelet/pods/67957741-7440-4dd3-9ddc-10dc3963b5f3/volumes" Oct 30 22:18:23.059465 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:23.059450 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a6d6940-a2e0-43fa-86af-0f84f6a22a39" path="/var/lib/kubelet/pods/9a6d6940-a2e0-43fa-86af-0f84f6a22a39/volumes" Oct 30 22:18:25.326092 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:18:25.326051 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:18:41.624320 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:41.624280 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:18:49.180057 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:49.180018 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx"] Oct 30 22:18:49.229149 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:49.229119 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b355441-ft2bx"] Oct 30 22:18:51.059276 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:18:51.059242 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f17319ef-ff41-4310-8126-4f50ec390d5f" path="/var/lib/kubelet/pods/f17319ef-ff41-4310-8126-4f50ec390d5f/volumes" Oct 30 22:18:55.321227 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:18:55.321188 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:19:07.008660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:07.008595 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq"] Oct 30 22:19:07.009188 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:07.008818 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" podUID="87f6d9bc-feca-46a3-925f-e3402a1d895c" containerName="istio-proxy" containerID="cri-o://8ab4780120d00a95318452be76e9c280b9fcda0051fb790f7a33adf475ceed1e" gracePeriod=30 Oct 30 22:19:07.500644 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:07.488954 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2"] Oct 30 22:19:07.500644 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:07.489221 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" podUID="a060b45e-4975-47da-ace0-889ca76b3424" containerName="istio-proxy" containerID="cri-o://1842a7f6fbef16976866fa7ddb6aec796ae616013f4a474b4bf53c916c531ee4" gracePeriod=30 Oct 30 22:19:08.584033 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.583999 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n"] Oct 30 22:19:08.585010 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.584988 4335 cpu_manager.go:414] "containerMap: removing container" podUID="67957741-7440-4dd3-9ddc-10dc3963b5f3" containerName="payload-processing" Oct 30 22:19:08.585010 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.585009 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67957741-7440-4dd3-9ddc-10dc3963b5f3" containerName="payload-processing" Oct 30 22:19:08.585218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.585052 4335 cpu_manager.go:414] "containerMap: removing container" podUID="9a6d6940-a2e0-43fa-86af-0f84f6a22a39" containerName="payload-pre-processing" Oct 30 22:19:08.585218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.585061 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6d6940-a2e0-43fa-86af-0f84f6a22a39" containerName="payload-pre-processing" Oct 30 22:19:08.585218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.585077 4335 cpu_manager.go:414] "containerMap: removing container" podUID="1122f4c8-84b1-46f0-825e-a51e2fb85b73" containerName="revoke-keys" Oct 30 22:19:08.585218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.585084 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1122f4c8-84b1-46f0-825e-a51e2fb85b73" containerName="revoke-keys" Oct 30 22:19:08.585434 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.585261 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="67957741-7440-4dd3-9ddc-10dc3963b5f3" containerName="payload-processing" Oct 30 22:19:08.585434 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.585280 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6d6940-a2e0-43fa-86af-0f84f6a22a39" containerName="payload-pre-processing" Oct 30 22:19:08.585434 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.585291 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="1122f4c8-84b1-46f0-825e-a51e2fb85b73" containerName="revoke-keys" Oct 30 22:19:08.589978 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.589954 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.643575 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.643535 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bdc5fe-openshift-default-dockercfg-b6mzp\"" Oct 30 22:19:08.698634 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.698587 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n"] Oct 30 22:19:08.729379 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.729339 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-workload-socket\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.729548 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.729485 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-istio-data\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.729709 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.729684 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-workload-certs\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.729780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.729765 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-istio-envoy\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.729849 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.729832 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-credential-socket\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.729916 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.729899 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvlgq\" (UniqueName: \"kubernetes.io/projected/967921e5-a600-4870-bb43-721aaffbad2d-kube-api-access-tvlgq\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.729978 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.729939 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967921e5-a600-4870-bb43-721aaffbad2d-istio-podinfo\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.729978 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.729965 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967921e5-a600-4870-bb43-721aaffbad2d-istio-token\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.730090 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.730062 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967921e5-a600-4870-bb43-721aaffbad2d-istiod-ca-cert\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.830975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.830946 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-workload-certs\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.831170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.831002 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-istio-envoy\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.831170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.831045 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-credential-socket\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.831170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.831065 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvlgq\" (UniqueName: \"kubernetes.io/projected/967921e5-a600-4870-bb43-721aaffbad2d-kube-api-access-tvlgq\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.831170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.831089 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967921e5-a600-4870-bb43-721aaffbad2d-istio-podinfo\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.831396 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.831299 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967921e5-a600-4870-bb43-721aaffbad2d-istio-token\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.831456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.831413 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-workload-certs\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.831456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.831431 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967921e5-a600-4870-bb43-721aaffbad2d-istiod-ca-cert\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.832572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.832214 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967921e5-a600-4870-bb43-721aaffbad2d-istiod-ca-cert\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.832572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.832319 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-workload-socket\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.832572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.832431 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-istio-data\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.832877 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.832681 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-workload-socket\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.832877 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.832786 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-istio-data\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.833251 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.833222 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-credential-socket\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.834104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.834039 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967921e5-a600-4870-bb43-721aaffbad2d-istio-podinfo\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.834265 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.834246 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-istio-envoy\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.914822 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.913196 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967921e5-a600-4870-bb43-721aaffbad2d-istio-token\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:08.919342 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:08.919150 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvlgq\" (UniqueName: \"kubernetes.io/projected/967921e5-a600-4870-bb43-721aaffbad2d-kube-api-access-tvlgq\") pod \"e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:09.205685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:09.205556 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:09.405118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:09.405079 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:19:09.405277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:09.405148 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" podUID="87f6d9bc-feca-46a3-925f-e3402a1d895c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:19:09.646712 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:19:09.646646 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967921e5_a600_4870_bb43_721aaffbad2d.slice/crio-55c24638ee7a5bc23ab8ccf692cb03f13943214df9c62650c84c389cc8e96fb0 WatchSource:0}: Error finding container 55c24638ee7a5bc23ab8ccf692cb03f13943214df9c62650c84c389cc8e96fb0: Status 404 returned error can't find the container with id 55c24638ee7a5bc23ab8ccf692cb03f13943214df9c62650c84c389cc8e96fb0 Oct 30 22:19:09.647878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:09.647810 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n"] Oct 30 22:19:09.650086 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:09.650054 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:19:09.650189 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:09.650121 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:19:09.650189 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:09.650155 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:19:10.432600 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:10.432560 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" event={"ID":"967921e5-a600-4870-bb43-721aaffbad2d","Type":"ContainerStarted","Data":"73f3917ebf1a46d077acfad490c34bc55de6a39d16003fda80d3eaebdb913436"} Oct 30 22:19:10.432795 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:10.432608 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" event={"ID":"967921e5-a600-4870-bb43-721aaffbad2d","Type":"ContainerStarted","Data":"55c24638ee7a5bc23ab8ccf692cb03f13943214df9c62650c84c389cc8e96fb0"} Oct 30 22:19:10.650860 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:10.650809 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" podStartSLOduration=2.650794057 podStartE2EDuration="2.650794057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:10.601200602 +0000 UTC m=+2083.701632335" watchObservedRunningTime="2026-10-30 22:19:10.650794057 +0000 UTC m=+2083.751225788" Oct 30 22:19:11.221993 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:11.195540 4335 patch_prober.go:29] interesting pod/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:19:11.221993 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:11.195649 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" podUID="a060b45e-4975-47da-ace0-889ca76b3424" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:19:11.224484 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:11.224452 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:11.249960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:11.249925 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:11.440466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:11.440436 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:11.441388 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:11.441365 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:19:12.285134 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.285111 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:19:12.409210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.409182 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-podinfo\") pod \"87f6d9bc-feca-46a3-925f-e3402a1d895c\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " Oct 30 22:19:12.409347 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.409216 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-credential-socket\") pod \"87f6d9bc-feca-46a3-925f-e3402a1d895c\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " Oct 30 22:19:12.409347 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.409243 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-workload-certs\") pod \"87f6d9bc-feca-46a3-925f-e3402a1d895c\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " Oct 30 22:19:12.409347 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.409261 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87f6d9bc-feca-46a3-925f-e3402a1d895c-kube-api-access-z84mn\" (UniqueName: \"kubernetes.io/projected/87f6d9bc-feca-46a3-925f-e3402a1d895c-kube-api-access-z84mn\") pod \"87f6d9bc-feca-46a3-925f-e3402a1d895c\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " Oct 30 22:19:12.409347 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.409302 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-data\") pod \"87f6d9bc-feca-46a3-925f-e3402a1d895c\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " Oct 30 22:19:12.409569 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.409367 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87f6d9bc-feca-46a3-925f-e3402a1d895c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87f6d9bc-feca-46a3-925f-e3402a1d895c-istiod-ca-cert\") pod \"87f6d9bc-feca-46a3-925f-e3402a1d895c\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " Oct 30 22:19:12.409569 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.409453 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-workload-socket\") pod \"87f6d9bc-feca-46a3-925f-e3402a1d895c\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " Oct 30 22:19:12.409569 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.409486 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-token\" (UniqueName: \"kubernetes.io/projected/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-token\") pod \"87f6d9bc-feca-46a3-925f-e3402a1d895c\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " Oct 30 22:19:12.409569 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.409482 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-credential-socket" pod "87f6d9bc-feca-46a3-925f-e3402a1d895c" (UID: "87f6d9bc-feca-46a3-925f-e3402a1d895c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:12.409569 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.409505 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-workload-certs" pod "87f6d9bc-feca-46a3-925f-e3402a1d895c" (UID: "87f6d9bc-feca-46a3-925f-e3402a1d895c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:12.409893 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.409686 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-envoy\") pod \"87f6d9bc-feca-46a3-925f-e3402a1d895c\" (UID: \"87f6d9bc-feca-46a3-925f-e3402a1d895c\") " Oct 30 22:19:12.409893 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.409789 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-workload-socket" pod "87f6d9bc-feca-46a3-925f-e3402a1d895c" (UID: "87f6d9bc-feca-46a3-925f-e3402a1d895c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:12.410062 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.409808 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87f6d9bc-feca-46a3-925f-e3402a1d895c-istiod-ca-cert" pod "87f6d9bc-feca-46a3-925f-e3402a1d895c" (UID: "87f6d9bc-feca-46a3-925f-e3402a1d895c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:12.410293 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.410270 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.410293 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.410282 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-data" pod "87f6d9bc-feca-46a3-925f-e3402a1d895c" (UID: "87f6d9bc-feca-46a3-925f-e3402a1d895c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:12.410507 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.410293 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.410507 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.410334 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87f6d9bc-feca-46a3-925f-e3402a1d895c-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.410507 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.410353 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.411770 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.411745 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-podinfo" pod "87f6d9bc-feca-46a3-925f-e3402a1d895c" (UID: "87f6d9bc-feca-46a3-925f-e3402a1d895c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:19:12.411855 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.411810 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-token" pod "87f6d9bc-feca-46a3-925f-e3402a1d895c" (UID: "87f6d9bc-feca-46a3-925f-e3402a1d895c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:12.411855 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.411825 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f6d9bc-feca-46a3-925f-e3402a1d895c-kube-api-access-z84mn" pod "87f6d9bc-feca-46a3-925f-e3402a1d895c" (UID: "87f6d9bc-feca-46a3-925f-e3402a1d895c"). InnerVolumeSpecName "kube-api-access-z84mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:12.412224 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.412209 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-envoy" pod "87f6d9bc-feca-46a3-925f-e3402a1d895c" (UID: "87f6d9bc-feca-46a3-925f-e3402a1d895c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:12.449531 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.449502 4335 generic.go:356] "Generic (PLEG): container finished" podID="87f6d9bc-feca-46a3-925f-e3402a1d895c" containerID="8ab4780120d00a95318452be76e9c280b9fcda0051fb790f7a33adf475ceed1e" exitCode=0 Oct 30 22:19:12.449672 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.449565 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" Oct 30 22:19:12.449672 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.449564 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" event={"ID":"87f6d9bc-feca-46a3-925f-e3402a1d895c","Type":"ContainerDied","Data":"8ab4780120d00a95318452be76e9c280b9fcda0051fb790f7a33adf475ceed1e"} Oct 30 22:19:12.449811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.449705 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq" event={"ID":"87f6d9bc-feca-46a3-925f-e3402a1d895c","Type":"ContainerDied","Data":"63dc384d09d9db0892ba0d514aa74319f66d7d65aea81bb5e4ee82199a808d65"} Oct 30 22:19:12.449811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.449734 4335 scope.go:122] "RemoveContainer" containerID="8ab4780120d00a95318452be76e9c280b9fcda0051fb790f7a33adf475ceed1e" Oct 30 22:19:12.465509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.465489 4335 scope.go:122] "RemoveContainer" containerID="8ab4780120d00a95318452be76e9c280b9fcda0051fb790f7a33adf475ceed1e" Oct 30 22:19:12.465867 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:19:12.465843 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab4780120d00a95318452be76e9c280b9fcda0051fb790f7a33adf475ceed1e\": container with ID starting with 8ab4780120d00a95318452be76e9c280b9fcda0051fb790f7a33adf475ceed1e not found: ID does not exist" containerID="8ab4780120d00a95318452be76e9c280b9fcda0051fb790f7a33adf475ceed1e" Oct 30 22:19:12.465957 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.465875 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab4780120d00a95318452be76e9c280b9fcda0051fb790f7a33adf475ceed1e"} err="failed to get container status \"8ab4780120d00a95318452be76e9c280b9fcda0051fb790f7a33adf475ceed1e\": rpc error: code = NotFound desc = could not find container \"8ab4780120d00a95318452be76e9c280b9fcda0051fb790f7a33adf475ceed1e\": container with ID starting with 8ab4780120d00a95318452be76e9c280b9fcda0051fb790f7a33adf475ceed1e not found: ID does not exist" Oct 30 22:19:12.512213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.512181 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.512348 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.512219 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.512348 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.512236 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.512348 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.512251 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z84mn\" (UniqueName: \"kubernetes.io/projected/87f6d9bc-feca-46a3-925f-e3402a1d895c-kube-api-access-z84mn\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.512348 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.512269 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87f6d9bc-feca-46a3-925f-e3402a1d895c-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.624425 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.624394 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq"] Oct 30 22:19:12.629383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.629352 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-h7trq"] Oct 30 22:19:12.780283 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.780260 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:19:12.922691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.918117 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-istio-envoy\") pod \"a060b45e-4975-47da-ace0-889ca76b3424\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " Oct 30 22:19:12.922691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.918280 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a060b45e-4975-47da-ace0-889ca76b3424-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a060b45e-4975-47da-ace0-889ca76b3424-istio-podinfo\") pod \"a060b45e-4975-47da-ace0-889ca76b3424\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " Oct 30 22:19:12.922691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.918320 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-istio-data\") pod \"a060b45e-4975-47da-ace0-889ca76b3424\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " Oct 30 22:19:12.922691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.918349 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a060b45e-4975-47da-ace0-889ca76b3424-kube-api-access-85v8m\" (UniqueName: \"kubernetes.io/projected/a060b45e-4975-47da-ace0-889ca76b3424-kube-api-access-85v8m\") pod \"a060b45e-4975-47da-ace0-889ca76b3424\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " Oct 30 22:19:12.922691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.918395 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a060b45e-4975-47da-ace0-889ca76b3424-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a060b45e-4975-47da-ace0-889ca76b3424-istiod-ca-cert\") pod \"a060b45e-4975-47da-ace0-889ca76b3424\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " Oct 30 22:19:12.922691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.918605 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-workload-certs\") pod \"a060b45e-4975-47da-ace0-889ca76b3424\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " Oct 30 22:19:12.922691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.918668 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-workload-socket\") pod \"a060b45e-4975-47da-ace0-889ca76b3424\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " Oct 30 22:19:12.922691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.918705 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a060b45e-4975-47da-ace0-889ca76b3424-istio-token\" (UniqueName: \"kubernetes.io/projected/a060b45e-4975-47da-ace0-889ca76b3424-istio-token\") pod \"a060b45e-4975-47da-ace0-889ca76b3424\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " Oct 30 22:19:12.922691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.918935 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-credential-socket\") pod \"a060b45e-4975-47da-ace0-889ca76b3424\" (UID: \"a060b45e-4975-47da-ace0-889ca76b3424\") " Oct 30 22:19:12.928318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.927477 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-workload-certs" pod "a060b45e-4975-47da-ace0-889ca76b3424" (UID: "a060b45e-4975-47da-ace0-889ca76b3424"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:12.928318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.927778 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a060b45e-4975-47da-ace0-889ca76b3424-istiod-ca-cert" pod "a060b45e-4975-47da-ace0-889ca76b3424" (UID: "a060b45e-4975-47da-ace0-889ca76b3424"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:12.928318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.927933 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-istio-data" pod "a060b45e-4975-47da-ace0-889ca76b3424" (UID: "a060b45e-4975-47da-ace0-889ca76b3424"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:12.928318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.928066 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-workload-socket" pod "a060b45e-4975-47da-ace0-889ca76b3424" (UID: "a060b45e-4975-47da-ace0-889ca76b3424"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:12.930264 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.930211 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-credential-socket" pod "a060b45e-4975-47da-ace0-889ca76b3424" (UID: "a060b45e-4975-47da-ace0-889ca76b3424"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:12.930969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.930938 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.931082 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.930976 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a060b45e-4975-47da-ace0-889ca76b3424-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.931082 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.930992 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.931082 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.931008 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.931082 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.931024 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.934467 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.933815 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a060b45e-4975-47da-ace0-889ca76b3424-kube-api-access-85v8m" pod "a060b45e-4975-47da-ace0-889ca76b3424" (UID: "a060b45e-4975-47da-ace0-889ca76b3424"). InnerVolumeSpecName "kube-api-access-85v8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:12.934467 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.934349 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a060b45e-4975-47da-ace0-889ca76b3424-istio-podinfo" pod "a060b45e-4975-47da-ace0-889ca76b3424" (UID: "a060b45e-4975-47da-ace0-889ca76b3424"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:19:12.934700 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.934480 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-istio-envoy" pod "a060b45e-4975-47da-ace0-889ca76b3424" (UID: "a060b45e-4975-47da-ace0-889ca76b3424"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:12.935133 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:12.935095 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a060b45e-4975-47da-ace0-889ca76b3424-istio-token" pod "a060b45e-4975-47da-ace0-889ca76b3424" (UID: "a060b45e-4975-47da-ace0-889ca76b3424"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:13.032863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:13.032832 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a060b45e-4975-47da-ace0-889ca76b3424-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:13.032863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:13.032861 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85v8m\" (UniqueName: \"kubernetes.io/projected/a060b45e-4975-47da-ace0-889ca76b3424-kube-api-access-85v8m\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:13.032863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:13.032873 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a060b45e-4975-47da-ace0-889ca76b3424-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:13.033107 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:13.032882 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a060b45e-4975-47da-ace0-889ca76b3424-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:13.064408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:13.064375 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87f6d9bc-feca-46a3-925f-e3402a1d895c" path="/var/lib/kubelet/pods/87f6d9bc-feca-46a3-925f-e3402a1d895c/volumes" Oct 30 22:19:13.459277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:13.459245 4335 generic.go:356] "Generic (PLEG): container finished" podID="a060b45e-4975-47da-ace0-889ca76b3424" containerID="1842a7f6fbef16976866fa7ddb6aec796ae616013f4a474b4bf53c916c531ee4" exitCode=0 Oct 30 22:19:13.459834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:13.459301 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" Oct 30 22:19:13.459834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:13.459332 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" event={"ID":"a060b45e-4975-47da-ace0-889ca76b3424","Type":"ContainerDied","Data":"1842a7f6fbef16976866fa7ddb6aec796ae616013f4a474b4bf53c916c531ee4"} Oct 30 22:19:13.459834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:13.459390 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2" event={"ID":"a060b45e-4975-47da-ace0-889ca76b3424","Type":"ContainerDied","Data":"c88564d2ff3a2431614c965aa4ee64265b27fb13b47adf4d67961bd0fa5f11ce"} Oct 30 22:19:13.459834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:13.459422 4335 scope.go:122] "RemoveContainer" containerID="1842a7f6fbef16976866fa7ddb6aec796ae616013f4a474b4bf53c916c531ee4" Oct 30 22:19:13.471103 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:13.471079 4335 scope.go:122] "RemoveContainer" containerID="1842a7f6fbef16976866fa7ddb6aec796ae616013f4a474b4bf53c916c531ee4" Oct 30 22:19:13.471389 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:19:13.471361 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1842a7f6fbef16976866fa7ddb6aec796ae616013f4a474b4bf53c916c531ee4\": container with ID starting with 1842a7f6fbef16976866fa7ddb6aec796ae616013f4a474b4bf53c916c531ee4 not found: ID does not exist" containerID="1842a7f6fbef16976866fa7ddb6aec796ae616013f4a474b4bf53c916c531ee4" Oct 30 22:19:13.471461 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:13.471396 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1842a7f6fbef16976866fa7ddb6aec796ae616013f4a474b4bf53c916c531ee4"} err="failed to get container status \"1842a7f6fbef16976866fa7ddb6aec796ae616013f4a474b4bf53c916c531ee4\": rpc error: code = NotFound desc = could not find container \"1842a7f6fbef16976866fa7ddb6aec796ae616013f4a474b4bf53c916c531ee4\": container with ID starting with 1842a7f6fbef16976866fa7ddb6aec796ae616013f4a474b4bf53c916c531ee4 not found: ID does not exist" Oct 30 22:19:13.631150 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:13.631110 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2"] Oct 30 22:19:13.680085 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:13.680055 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-gdcv2"] Oct 30 22:19:15.060202 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:15.060155 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a060b45e-4975-47da-ace0-889ca76b3424" path="/var/lib/kubelet/pods/a060b45e-4975-47da-ace0-889ca76b3424/volumes" Oct 30 22:19:18.486428 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.486394 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj"] Oct 30 22:19:18.487104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.487088 4335 cpu_manager.go:414] "containerMap: removing container" podUID="87f6d9bc-feca-46a3-925f-e3402a1d895c" containerName="istio-proxy" Oct 30 22:19:18.487152 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.487106 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f6d9bc-feca-46a3-925f-e3402a1d895c" containerName="istio-proxy" Oct 30 22:19:18.487152 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.487127 4335 cpu_manager.go:414] "containerMap: removing container" podUID="a060b45e-4975-47da-ace0-889ca76b3424" containerName="istio-proxy" Oct 30 22:19:18.487152 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.487133 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a060b45e-4975-47da-ace0-889ca76b3424" containerName="istio-proxy" Oct 30 22:19:18.487255 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.487246 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f6d9bc-feca-46a3-925f-e3402a1d895c" containerName="istio-proxy" Oct 30 22:19:18.487292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.487259 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="a060b45e-4975-47da-ace0-889ca76b3424" containerName="istio-proxy" Oct 30 22:19:18.491064 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.491046 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj" Oct 30 22:19:18.551258 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.551227 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj"] Oct 30 22:19:18.575974 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.575938 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbjnl\" (UniqueName: \"kubernetes.io/projected/b761d5cf-8cdd-42b8-bac9-b6724e06bac5-kube-api-access-vbjnl\") pod \"payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj\" (UID: \"b761d5cf-8cdd-42b8-bac9-b6724e06bac5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj" Oct 30 22:19:18.576386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.576358 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b761d5cf-8cdd-42b8-bac9-b6724e06bac5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj\" (UID: \"b761d5cf-8cdd-42b8-bac9-b6724e06bac5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj" Oct 30 22:19:18.615963 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.615873 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-bdc5fe\"" Oct 30 22:19:18.616352 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.616335 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-bdc5fe-dockercfg-szm4t\"" Oct 30 22:19:18.635658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.625927 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg"] Oct 30 22:19:18.636889 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.635960 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg" Oct 30 22:19:18.686634 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.684415 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8cde58-7add-4dac-97c6-b0ec3482832c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg\" (UID: \"ba8cde58-7add-4dac-97c6-b0ec3482832c\") " pod="openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg" Oct 30 22:19:18.686634 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.684586 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbjnl\" (UniqueName: \"kubernetes.io/projected/b761d5cf-8cdd-42b8-bac9-b6724e06bac5-kube-api-access-vbjnl\") pod \"payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj\" (UID: \"b761d5cf-8cdd-42b8-bac9-b6724e06bac5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj" Oct 30 22:19:18.686634 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.684644 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf56h\" (UniqueName: \"kubernetes.io/projected/ba8cde58-7add-4dac-97c6-b0ec3482832c-kube-api-access-cf56h\") pod \"payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg\" (UID: \"ba8cde58-7add-4dac-97c6-b0ec3482832c\") " pod="openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg" Oct 30 22:19:18.686634 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.684703 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b761d5cf-8cdd-42b8-bac9-b6724e06bac5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj\" (UID: \"b761d5cf-8cdd-42b8-bac9-b6724e06bac5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj" Oct 30 22:19:18.698689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.687839 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b761d5cf-8cdd-42b8-bac9-b6724e06bac5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj\" (UID: \"b761d5cf-8cdd-42b8-bac9-b6724e06bac5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj" Oct 30 22:19:18.782681 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.777234 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbjnl\" (UniqueName: \"kubernetes.io/projected/b761d5cf-8cdd-42b8-bac9-b6724e06bac5-kube-api-access-vbjnl\") pod \"payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj\" (UID: \"b761d5cf-8cdd-42b8-bac9-b6724e06bac5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj" Oct 30 22:19:18.787088 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.787052 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf56h\" (UniqueName: \"kubernetes.io/projected/ba8cde58-7add-4dac-97c6-b0ec3482832c-kube-api-access-cf56h\") pod \"payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg\" (UID: \"ba8cde58-7add-4dac-97c6-b0ec3482832c\") " pod="openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg" Oct 30 22:19:18.787495 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.787390 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8cde58-7add-4dac-97c6-b0ec3482832c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg\" (UID: \"ba8cde58-7add-4dac-97c6-b0ec3482832c\") " pod="openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg" Oct 30 22:19:18.788441 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.788417 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8cde58-7add-4dac-97c6-b0ec3482832c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg\" (UID: \"ba8cde58-7add-4dac-97c6-b0ec3482832c\") " pod="openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg" Oct 30 22:19:18.794711 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.794690 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg"] Oct 30 22:19:18.829633 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.829114 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj" Oct 30 22:19:18.848587 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:18.848556 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf56h\" (UniqueName: \"kubernetes.io/projected/ba8cde58-7add-4dac-97c6-b0ec3482832c-kube-api-access-cf56h\") pod \"payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg\" (UID: \"ba8cde58-7add-4dac-97c6-b0ec3482832c\") " pod="openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg" Oct 30 22:19:19.084156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:19.084125 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg" Oct 30 22:19:19.230555 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:19.230328 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj"] Oct 30 22:19:19.482810 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:19:19.482781 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8cde58_7add_4dac_97c6_b0ec3482832c.slice/crio-a85e3e8ff02f6b0d2591b17687493db5d8d030692d212a4b2f67e54d89f59974 WatchSource:0}: Error finding container a85e3e8ff02f6b0d2591b17687493db5d8d030692d212a4b2f67e54d89f59974: Status 404 returned error can't find the container with id a85e3e8ff02f6b0d2591b17687493db5d8d030692d212a4b2f67e54d89f59974 Oct 30 22:19:19.483116 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:19.483095 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg"] Oct 30 22:19:19.556673 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:19.556632 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj" event={"ID":"b761d5cf-8cdd-42b8-bac9-b6724e06bac5","Type":"ContainerStarted","Data":"fa8bb92d0f1ef2acb646be1926921974ffa99883e6193f8aaf76e53336c370bb"} Oct 30 22:19:19.557049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:19.556704 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj" event={"ID":"b761d5cf-8cdd-42b8-bac9-b6724e06bac5","Type":"ContainerStarted","Data":"1e4b1679c7588acd989aa9cf87e76770fe9828dc96a1adf908018467508caf6e"} Oct 30 22:19:19.557049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:19.556923 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj" Oct 30 22:19:19.559766 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:19.559664 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg" event={"ID":"ba8cde58-7add-4dac-97c6-b0ec3482832c","Type":"ContainerStarted","Data":"a85e3e8ff02f6b0d2591b17687493db5d8d030692d212a4b2f67e54d89f59974"} Oct 30 22:19:19.675057 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:19.675009 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj" podStartSLOduration=1.6749956670000001 podStartE2EDuration="1.674995667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:19.672381343 +0000 UTC m=+2092.772813074" watchObservedRunningTime="2026-10-30 22:19:19.674995667 +0000 UTC m=+2092.775427398" Oct 30 22:19:20.573196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:20.573158 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg" event={"ID":"ba8cde58-7add-4dac-97c6-b0ec3482832c","Type":"ContainerStarted","Data":"ebf75bd1e8b4e44dc502eac9bfeeff160fb5420fadbea139dd2e48cf9330e36d"} Oct 30 22:19:20.573748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:20.573564 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg" Oct 30 22:19:20.692371 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:20.692316 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg" podStartSLOduration=2.692301563 podStartE2EDuration="2.692301563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:20.690573804 +0000 UTC m=+2093.791005536" watchObservedRunningTime="2026-10-30 22:19:20.692301563 +0000 UTC m=+2093.792733294" Oct 30 22:19:23.611964 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:23.611920 4335 generic.go:356] "Generic (PLEG): container finished" podID="e7f63a86-2ce4-45ad-8f6a-38025710f1f5" containerID="797a88611d5a92c6fdd8203ced121e2795c14e6c30a5a18512daec383c220d05" exitCode=0 Oct 30 22:19:23.612506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:23.611991 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" event={"ID":"e7f63a86-2ce4-45ad-8f6a-38025710f1f5","Type":"ContainerDied","Data":"797a88611d5a92c6fdd8203ced121e2795c14e6c30a5a18512daec383c220d05"} Oct 30 22:19:23.612506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:23.612374 4335 scope.go:122] "RemoveContainer" containerID="797a88611d5a92c6fdd8203ced121e2795c14e6c30a5a18512daec383c220d05" Oct 30 22:19:23.614003 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:23.613974 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:19:23.614116 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:23.614046 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:19:23.614116 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:23.614077 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:19:24.651226 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:24.651197 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" event={"ID":"e7f63a86-2ce4-45ad-8f6a-38025710f1f5","Type":"ContainerStarted","Data":"5642d47bd7500a209cf89862e77d3baee4c4bbc672c30398736b944b1374d51a"} Oct 30 22:19:24.752679 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:24.752632 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:19:24.752864 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:24.752841 4335 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 22:19:24.752956 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:24.752934 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" podUID="e7f63a86-2ce4-45ad-8f6a-38025710f1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 22:19:25.350558 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:19:25.350511 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:19:25.753234 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:25.753137 4335 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 22:19:25.753234 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:25.753194 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" podUID="e7f63a86-2ce4-45ad-8f6a-38025710f1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 22:19:26.753093 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:26.753057 4335 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 22:19:26.753268 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:26.753122 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" podUID="e7f63a86-2ce4-45ad-8f6a-38025710f1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 22:19:27.753175 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:27.753142 4335 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 22:19:27.753363 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:27.753196 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" podUID="e7f63a86-2ce4-45ad-8f6a-38025710f1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 22:19:28.233901 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:28.233877 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:19:28.358009 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:28.357979 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:19:28.752934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:28.752885 4335 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 22:19:28.753145 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:28.753004 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" podUID="e7f63a86-2ce4-45ad-8f6a-38025710f1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 22:19:29.186533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:29.186504 4335 scope.go:122] "RemoveContainer" containerID="3efe763906edcb384821872f34b635eed10fc99ff2a6311aa4cf4affb106c561" Oct 30 22:19:29.216931 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:29.215739 4335 scope.go:122] "RemoveContainer" containerID="0f90f7741569279ea0f2f2ef510bb38d5ebb5e26870581ce8cd5f53261cd4dbc" Oct 30 22:19:29.234345 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:29.234280 4335 scope.go:122] "RemoveContainer" containerID="344fe711fc609654437a2a71a769ddce1f8ad3b712e6bdb549ed6d212f2c20ad" Oct 30 22:19:29.247198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:29.247180 4335 scope.go:122] "RemoveContainer" containerID="bbeb5b28aaa6b292f139c64321f2bcd98355d0218ff38c9f51b17722165d1b71" Oct 30 22:19:29.752455 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:29.752419 4335 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 22:19:29.752680 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:29.752488 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" podUID="e7f63a86-2ce4-45ad-8f6a-38025710f1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 22:19:30.169089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:30.169062 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:19:30.575628 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:30.575590 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj" Oct 30 22:19:30.757384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:30.757355 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:19:30.758491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:30.758468 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:19:31.593485 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:31.593456 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg" Oct 30 22:19:35.532289 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.532255 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6"] Oct 30 22:19:35.539630 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.539586 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.602008 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.601971 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2d3647-openshift-default-dockercfg-8ztdb\"" Oct 30 22:19:35.731657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.730322 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-workload-certs\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.731657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.730417 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-istio-data\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.731657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.730473 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpkzr\" (UniqueName: \"kubernetes.io/projected/350244ea-9e66-4e83-84ff-ef81766feab1-kube-api-access-zpkzr\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.731657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.730539 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/350244ea-9e66-4e83-84ff-ef81766feab1-istio-token\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.731657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.730584 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/350244ea-9e66-4e83-84ff-ef81766feab1-istio-podinfo\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.731657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.730729 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/350244ea-9e66-4e83-84ff-ef81766feab1-istiod-ca-cert\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.731657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.730774 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-credential-socket\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.731657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.730808 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-workload-socket\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.731657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.730908 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-istio-envoy\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.783120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.783021 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6"] Oct 30 22:19:35.832505 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.832465 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-workload-certs\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.833515 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.833476 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-istio-data\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.833786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.833768 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpkzr\" (UniqueName: \"kubernetes.io/projected/350244ea-9e66-4e83-84ff-ef81766feab1-kube-api-access-zpkzr\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.834600 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.834579 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/350244ea-9e66-4e83-84ff-ef81766feab1-istio-token\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.835105 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.835071 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/350244ea-9e66-4e83-84ff-ef81766feab1-istio-podinfo\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.835361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.835324 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/350244ea-9e66-4e83-84ff-ef81766feab1-istiod-ca-cert\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.835509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.835495 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-credential-socket\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.835991 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.835959 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-workload-socket\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.836179 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.836164 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-istio-envoy\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.836725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.833136 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-workload-certs\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.836871 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.834145 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-istio-data\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.837602 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.837564 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-credential-socket\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.838600 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.838320 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/350244ea-9e66-4e83-84ff-ef81766feab1-istiod-ca-cert\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.840339 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.838528 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-workload-socket\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.859487 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.859456 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-istio-envoy\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.881537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.877743 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/350244ea-9e66-4e83-84ff-ef81766feab1-istio-podinfo\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.895531 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.895502 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpkzr\" (UniqueName: \"kubernetes.io/projected/350244ea-9e66-4e83-84ff-ef81766feab1-kube-api-access-zpkzr\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:35.896435 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:35.896417 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/350244ea-9e66-4e83-84ff-ef81766feab1-istio-token\") pod \"e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:36.187151 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:36.187120 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:36.632853 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:36.632816 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6"] Oct 30 22:19:36.641940 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:19:36.641908 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350244ea_9e66_4e83_84ff_ef81766feab1.slice/crio-e606f2d6d0e57bc6093ba225dfaf76621033b64580f72afe8c51aafa1e72dfbd WatchSource:0}: Error finding container e606f2d6d0e57bc6093ba225dfaf76621033b64580f72afe8c51aafa1e72dfbd: Status 404 returned error can't find the container with id e606f2d6d0e57bc6093ba225dfaf76621033b64580f72afe8c51aafa1e72dfbd Oct 30 22:19:36.645047 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:36.645015 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:19:36.645121 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:36.645082 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:19:36.645121 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:36.645108 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:19:36.827346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:36.827311 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" event={"ID":"350244ea-9e66-4e83-84ff-ef81766feab1","Type":"ContainerStarted","Data":"1dd72879d2c30c356158d8252ae367488e3cd84f0e38cbadf5946c37f3a434e5"} Oct 30 22:19:36.827552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:36.827373 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" event={"ID":"350244ea-9e66-4e83-84ff-ef81766feab1","Type":"ContainerStarted","Data":"e606f2d6d0e57bc6093ba225dfaf76621033b64580f72afe8c51aafa1e72dfbd"} Oct 30 22:19:36.951324 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:36.951140 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" podStartSLOduration=1.951123489 podStartE2EDuration="1.951123489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:36.950724376 +0000 UTC m=+2110.051156111" watchObservedRunningTime="2026-10-30 22:19:36.951123489 +0000 UTC m=+2110.051555221" Oct 30 22:19:37.188129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:37.188090 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:37.194108 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:37.194086 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:37.411185 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:37.411153 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp"] Oct 30 22:19:37.411510 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:37.411358 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" podUID="5c5448fc-7500-43b9-abd3-f9a8c603fbd0" containerName="istio-proxy" containerID="cri-o://21cfb229c638e174dbbcaab01daebd6afe409b2fd6c987b602f2d2592bd9c2f6" gracePeriod=30 Oct 30 22:19:37.837200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:37.837167 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:37.837987 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:37.837970 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:19:42.886581 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:42.886544 4335 generic.go:356] "Generic (PLEG): container finished" podID="5c5448fc-7500-43b9-abd3-f9a8c603fbd0" containerID="21cfb229c638e174dbbcaab01daebd6afe409b2fd6c987b602f2d2592bd9c2f6" exitCode=0 Oct 30 22:19:42.886971 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:42.886655 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" event={"ID":"5c5448fc-7500-43b9-abd3-f9a8c603fbd0","Type":"ContainerDied","Data":"21cfb229c638e174dbbcaab01daebd6afe409b2fd6c987b602f2d2592bd9c2f6"} Oct 30 22:19:43.105773 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.105750 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:19:43.268445 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.268364 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istiod-ca-cert\") pod \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " Oct 30 22:19:43.268643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.268509 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-kube-api-access-h8fjp\" (UniqueName: \"kubernetes.io/projected/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-kube-api-access-h8fjp\") pod \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " Oct 30 22:19:43.268643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.268549 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-token\" (UniqueName: \"kubernetes.io/projected/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-token\") pod \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " Oct 30 22:19:43.268797 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.268646 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-workload-certs\") pod \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " Oct 30 22:19:43.268797 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.268711 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-podinfo\") pod \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " Oct 30 22:19:43.268797 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.268731 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istiod-ca-cert" pod "5c5448fc-7500-43b9-abd3-f9a8c603fbd0" (UID: "5c5448fc-7500-43b9-abd3-f9a8c603fbd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:43.268797 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.268792 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-workload-socket\") pod \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " Oct 30 22:19:43.269015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.268831 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-envoy\") pod \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " Oct 30 22:19:43.269015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.268893 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-data\") pod \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " Oct 30 22:19:43.269015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.268923 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-workload-certs" pod "5c5448fc-7500-43b9-abd3-f9a8c603fbd0" (UID: "5c5448fc-7500-43b9-abd3-f9a8c603fbd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:43.269190 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.269031 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-workload-socket" pod "5c5448fc-7500-43b9-abd3-f9a8c603fbd0" (UID: "5c5448fc-7500-43b9-abd3-f9a8c603fbd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:43.269190 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.269181 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-data" pod "5c5448fc-7500-43b9-abd3-f9a8c603fbd0" (UID: "5c5448fc-7500-43b9-abd3-f9a8c603fbd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:43.269319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.269299 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-credential-socket\") pod \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\" (UID: \"5c5448fc-7500-43b9-abd3-f9a8c603fbd0\") " Oct 30 22:19:43.269890 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.269862 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:43.269890 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.269889 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:43.270076 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.269905 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:43.270076 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.269919 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:43.270076 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.269927 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-credential-socket" pod "5c5448fc-7500-43b9-abd3-f9a8c603fbd0" (UID: "5c5448fc-7500-43b9-abd3-f9a8c603fbd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:43.271051 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.271027 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-envoy" pod "5c5448fc-7500-43b9-abd3-f9a8c603fbd0" (UID: "5c5448fc-7500-43b9-abd3-f9a8c603fbd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:43.271137 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.271125 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-podinfo" pod "5c5448fc-7500-43b9-abd3-f9a8c603fbd0" (UID: "5c5448fc-7500-43b9-abd3-f9a8c603fbd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:19:43.271952 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.271930 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-kube-api-access-h8fjp" pod "5c5448fc-7500-43b9-abd3-f9a8c603fbd0" (UID: "5c5448fc-7500-43b9-abd3-f9a8c603fbd0"). InnerVolumeSpecName "kube-api-access-h8fjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:43.273196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.273179 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-token" pod "5c5448fc-7500-43b9-abd3-f9a8c603fbd0" (UID: "5c5448fc-7500-43b9-abd3-f9a8c603fbd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:43.371822 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.371791 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8fjp\" (UniqueName: \"kubernetes.io/projected/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-kube-api-access-h8fjp\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:43.371822 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.371821 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:43.372019 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.371831 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:43.372019 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.371841 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:43.372019 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.371850 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5448fc-7500-43b9-abd3-f9a8c603fbd0-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:43.629343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.629309 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs"] Oct 30 22:19:43.630168 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.630146 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5c5448fc-7500-43b9-abd3-f9a8c603fbd0" containerName="istio-proxy" Oct 30 22:19:43.630284 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.630169 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c5448fc-7500-43b9-abd3-f9a8c603fbd0" containerName="istio-proxy" Oct 30 22:19:43.630383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.630369 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c5448fc-7500-43b9-abd3-f9a8c603fbd0" containerName="istio-proxy" Oct 30 22:19:43.634162 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.634141 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs" Oct 30 22:19:43.687154 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.687128 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2d3647-dockercfg-kkqrg\"" Oct 30 22:19:43.687310 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.687219 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2d3647\"" Oct 30 22:19:43.692577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.692549 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs"] Oct 30 22:19:43.802656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.788956 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbecb7f4-f382-4797-aaca-f4c362171cd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs\" (UID: \"dbecb7f4-f382-4797-aaca-f4c362171cd9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs" Oct 30 22:19:43.802656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.789016 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmrzd\" (UniqueName: \"kubernetes.io/projected/dbecb7f4-f382-4797-aaca-f4c362171cd9-kube-api-access-qmrzd\") pod \"payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs\" (UID: \"dbecb7f4-f382-4797-aaca-f4c362171cd9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs" Oct 30 22:19:43.882396 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.882135 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn"] Oct 30 22:19:43.938884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.937877 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbecb7f4-f382-4797-aaca-f4c362171cd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs\" (UID: \"dbecb7f4-f382-4797-aaca-f4c362171cd9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs" Oct 30 22:19:43.938884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.937939 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmrzd\" (UniqueName: \"kubernetes.io/projected/dbecb7f4-f382-4797-aaca-f4c362171cd9-kube-api-access-qmrzd\") pod \"payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs\" (UID: \"dbecb7f4-f382-4797-aaca-f4c362171cd9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs" Oct 30 22:19:43.939653 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.939074 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbecb7f4-f382-4797-aaca-f4c362171cd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs\" (UID: \"dbecb7f4-f382-4797-aaca-f4c362171cd9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs" Oct 30 22:19:43.954251 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.954168 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn" Oct 30 22:19:43.955794 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.955349 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn"] Oct 30 22:19:43.966361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.966332 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" event={"ID":"5c5448fc-7500-43b9-abd3-f9a8c603fbd0","Type":"ContainerDied","Data":"b648a5066b1fb5e14c2fff22975b7a76046233e89dd226e13184e47695d6a6f5"} Oct 30 22:19:43.966361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.966356 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp" Oct 30 22:19:43.966533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:43.966392 4335 scope.go:122] "RemoveContainer" containerID="21cfb229c638e174dbbcaab01daebd6afe409b2fd6c987b602f2d2592bd9c2f6" Oct 30 22:19:44.001396 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:44.001371 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmrzd\" (UniqueName: \"kubernetes.io/projected/dbecb7f4-f382-4797-aaca-f4c362171cd9-kube-api-access-qmrzd\") pod \"payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs\" (UID: \"dbecb7f4-f382-4797-aaca-f4c362171cd9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs" Oct 30 22:19:44.143118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:44.143047 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69txr\" (UniqueName: \"kubernetes.io/projected/7d13516d-0abb-4663-a016-45a93612d102-kube-api-access-69txr\") pod \"payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn\" (UID: \"7d13516d-0abb-4663-a016-45a93612d102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn" Oct 30 22:19:44.143118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:44.143111 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d13516d-0abb-4663-a016-45a93612d102-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn\" (UID: \"7d13516d-0abb-4663-a016-45a93612d102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn" Oct 30 22:19:44.215506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:44.215476 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp"] Oct 30 22:19:44.219852 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:44.219827 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-v6ppp"] Oct 30 22:19:44.244841 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:44.244804 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69txr\" (UniqueName: \"kubernetes.io/projected/7d13516d-0abb-4663-a016-45a93612d102-kube-api-access-69txr\") pod \"payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn\" (UID: \"7d13516d-0abb-4663-a016-45a93612d102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn" Oct 30 22:19:44.244944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:44.244854 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d13516d-0abb-4663-a016-45a93612d102-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn\" (UID: \"7d13516d-0abb-4663-a016-45a93612d102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn" Oct 30 22:19:44.245496 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:44.245478 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d13516d-0abb-4663-a016-45a93612d102-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn\" (UID: \"7d13516d-0abb-4663-a016-45a93612d102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn" Oct 30 22:19:44.251475 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:44.251448 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs" Oct 30 22:19:44.319782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:44.315830 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69txr\" (UniqueName: \"kubernetes.io/projected/7d13516d-0abb-4663-a016-45a93612d102-kube-api-access-69txr\") pod \"payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn\" (UID: \"7d13516d-0abb-4663-a016-45a93612d102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn" Oct 30 22:19:44.575994 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:44.575970 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn" Oct 30 22:19:44.644006 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:44.643980 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs"] Oct 30 22:19:44.646416 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:19:44.646390 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbecb7f4_f382_4797_aaca_f4c362171cd9.slice/crio-c24c7959c2a4f67ccd4c259420e00075839184386ae958a46780e2999dc6a8b2 WatchSource:0}: Error finding container c24c7959c2a4f67ccd4c259420e00075839184386ae958a46780e2999dc6a8b2: Status 404 returned error can't find the container with id c24c7959c2a4f67ccd4c259420e00075839184386ae958a46780e2999dc6a8b2 Oct 30 22:19:44.966913 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:44.966880 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn"] Oct 30 22:19:44.968206 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:19:44.968180 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d13516d_0abb_4663_a016_45a93612d102.slice/crio-bb3730cc629081595847b366810b31bebd2b692f42b50a07cb7f224a2430bbdc WatchSource:0}: Error finding container bb3730cc629081595847b366810b31bebd2b692f42b50a07cb7f224a2430bbdc: Status 404 returned error can't find the container with id bb3730cc629081595847b366810b31bebd2b692f42b50a07cb7f224a2430bbdc Oct 30 22:19:44.976520 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:44.976490 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs" event={"ID":"dbecb7f4-f382-4797-aaca-f4c362171cd9","Type":"ContainerStarted","Data":"4b5658a5b131b2a598b41a6c17dd2a07e01ad35bda6b9fbf986569c35e7e94ab"} Oct 30 22:19:44.976664 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:44.976546 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs" event={"ID":"dbecb7f4-f382-4797-aaca-f4c362171cd9","Type":"ContainerStarted","Data":"c24c7959c2a4f67ccd4c259420e00075839184386ae958a46780e2999dc6a8b2"} Oct 30 22:19:44.976744 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:44.976733 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs" Oct 30 22:19:45.061273 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:45.061244 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c5448fc-7500-43b9-abd3-f9a8c603fbd0" path="/var/lib/kubelet/pods/5c5448fc-7500-43b9-abd3-f9a8c603fbd0/volumes" Oct 30 22:19:45.094817 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:45.094772 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs" podStartSLOduration=2.094757519 podStartE2EDuration="2.094757519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:45.091603317 +0000 UTC m=+2118.192035048" watchObservedRunningTime="2026-10-30 22:19:45.094757519 +0000 UTC m=+2118.195189251" Oct 30 22:19:45.988538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:45.988499 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn" event={"ID":"7d13516d-0abb-4663-a016-45a93612d102","Type":"ContainerStarted","Data":"1e6a5d668425a9206f48bb960101ef328d87974b3f8c169fa63484ad8b0f6c58"} Oct 30 22:19:45.989016 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:45.988562 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn" event={"ID":"7d13516d-0abb-4663-a016-45a93612d102","Type":"ContainerStarted","Data":"bb3730cc629081595847b366810b31bebd2b692f42b50a07cb7f224a2430bbdc"} Oct 30 22:19:45.989016 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:45.988732 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn" Oct 30 22:19:46.110935 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:46.110881 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn" podStartSLOduration=3.110863758 podStartE2EDuration="3.110863758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:46.103679823 +0000 UTC m=+2119.204111556" watchObservedRunningTime="2026-10-30 22:19:46.110863758 +0000 UTC m=+2119.211295493" Oct 30 22:19:53.726360 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:53.726325 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:19:55.340793 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:19:55.340753 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:19:55.991933 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:55.991897 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs" Oct 30 22:19:57.023811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:19:57.023774 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn" Oct 30 22:20:04.142794 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:04.142764 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5"] Oct 30 22:20:04.146745 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:04.146723 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5" Oct 30 22:20:04.205901 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:04.205875 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5"] Oct 30 22:20:04.247669 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:04.247641 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:20:04.247669 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:04.247659 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:20:04.247913 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:04.247717 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:20:04.248574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:04.248554 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:20:04.251654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:04.251599 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26693c1f-2409-4b8b-94ed-c52e46123c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5feaf50-cf2v5\" (UID: \"26693c1f-2409-4b8b-94ed-c52e46123c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5" Oct 30 22:20:04.251804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:04.251784 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6x5q\" (UniqueName: \"kubernetes.io/projected/26693c1f-2409-4b8b-94ed-c52e46123c3e-kube-api-access-h6x5q\") pod \"maas-api-revoke-sub-f5feaf50-cf2v5\" (UID: \"26693c1f-2409-4b8b-94ed-c52e46123c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5" Oct 30 22:20:04.252051 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:04.252027 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26693c1f-2409-4b8b-94ed-c52e46123c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5feaf50-cf2v5\" (UID: \"26693c1f-2409-4b8b-94ed-c52e46123c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5" Oct 30 22:20:04.353815 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:04.353778 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26693c1f-2409-4b8b-94ed-c52e46123c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5feaf50-cf2v5\" (UID: \"26693c1f-2409-4b8b-94ed-c52e46123c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5" Oct 30 22:20:04.353952 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:04.353938 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26693c1f-2409-4b8b-94ed-c52e46123c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5feaf50-cf2v5\" (UID: \"26693c1f-2409-4b8b-94ed-c52e46123c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5" Oct 30 22:20:04.354049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:04.354035 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6x5q\" (UniqueName: \"kubernetes.io/projected/26693c1f-2409-4b8b-94ed-c52e46123c3e-kube-api-access-h6x5q\") pod \"maas-api-revoke-sub-f5feaf50-cf2v5\" (UID: \"26693c1f-2409-4b8b-94ed-c52e46123c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5" Oct 30 22:20:04.354445 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:04.354417 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26693c1f-2409-4b8b-94ed-c52e46123c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5feaf50-cf2v5\" (UID: \"26693c1f-2409-4b8b-94ed-c52e46123c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5" Oct 30 22:20:04.356288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:04.356263 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26693c1f-2409-4b8b-94ed-c52e46123c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5feaf50-cf2v5\" (UID: \"26693c1f-2409-4b8b-94ed-c52e46123c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5" Oct 30 22:20:04.415007 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:04.414928 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6x5q\" (UniqueName: \"kubernetes.io/projected/26693c1f-2409-4b8b-94ed-c52e46123c3e-kube-api-access-h6x5q\") pod \"maas-api-revoke-sub-f5feaf50-cf2v5\" (UID: \"26693c1f-2409-4b8b-94ed-c52e46123c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5" Oct 30 22:20:04.462740 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:04.462713 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5" Oct 30 22:20:04.845647 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:20:04.845584 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26693c1f_2409_4b8b_94ed_c52e46123c3e.slice/crio-ddb846a0fd7fa2255e4e112a9259b9e8deb9b36b82256e07b39bf27a0de79166 WatchSource:0}: Error finding container ddb846a0fd7fa2255e4e112a9259b9e8deb9b36b82256e07b39bf27a0de79166: Status 404 returned error can't find the container with id ddb846a0fd7fa2255e4e112a9259b9e8deb9b36b82256e07b39bf27a0de79166 Oct 30 22:20:04.846261 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:04.846237 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5"] Oct 30 22:20:05.331212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:05.331174 4335 generic.go:356] "Generic (PLEG): container finished" podID="5e3c2933-2586-4001-aa99-c18c1a6d44bb" containerID="868c00326a18b5e45b5760dde9dfd0fb89c3bf7d9aff85cecde3eb47a5b3350a" exitCode=0 Oct 30 22:20:05.331860 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:05.331833 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" event={"ID":"5e3c2933-2586-4001-aa99-c18c1a6d44bb","Type":"ContainerDied","Data":"868c00326a18b5e45b5760dde9dfd0fb89c3bf7d9aff85cecde3eb47a5b3350a"} Oct 30 22:20:05.333595 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:05.333337 4335 scope.go:122] "RemoveContainer" containerID="868c00326a18b5e45b5760dde9dfd0fb89c3bf7d9aff85cecde3eb47a5b3350a" Oct 30 22:20:05.336951 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:05.336923 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5" event={"ID":"26693c1f-2409-4b8b-94ed-c52e46123c3e","Type":"ContainerStarted","Data":"6cd0eb93248c093fa5c408623500ecdaca159c9dad58bfcae72782c5a763fb49"} Oct 30 22:20:05.337728 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:05.337694 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5" event={"ID":"26693c1f-2409-4b8b-94ed-c52e46123c3e","Type":"ContainerStarted","Data":"ddb846a0fd7fa2255e4e112a9259b9e8deb9b36b82256e07b39bf27a0de79166"} Oct 30 22:20:05.338708 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:05.338674 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:20:05.338918 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:05.338899 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:20:05.339076 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:05.339059 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:20:05.573744 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:05.573680 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5" podStartSLOduration=1.573659651 podStartE2EDuration="1.573659651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:05.570527172 +0000 UTC m=+2138.670958904" watchObservedRunningTime="2026-10-30 22:20:05.573659651 +0000 UTC m=+2138.674091384" Oct 30 22:20:06.351887 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:06.351849 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" event={"ID":"5e3c2933-2586-4001-aa99-c18c1a6d44bb","Type":"ContainerStarted","Data":"11fce6f669b7c04c675f8f922eb71fcc58e0adfab32b23bbaa5fa495eebfa8f1"} Oct 30 22:20:06.353887 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:06.353855 4335 generic.go:356] "Generic (PLEG): container finished" podID="26693c1f-2409-4b8b-94ed-c52e46123c3e" containerID="6cd0eb93248c093fa5c408623500ecdaca159c9dad58bfcae72782c5a763fb49" exitCode=0 Oct 30 22:20:06.354041 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:06.353933 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5" event={"ID":"26693c1f-2409-4b8b-94ed-c52e46123c3e","Type":"ContainerDied","Data":"6cd0eb93248c093fa5c408623500ecdaca159c9dad58bfcae72782c5a763fb49"} Oct 30 22:20:06.853990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:06.852686 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6"] Oct 30 22:20:06.853990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:06.852931 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" podUID="63c520ab-1d01-497c-9ac9-a10bfd284e54" containerName="istio-proxy" containerID="cri-o://b9de59cda9a88634dc048d8d522d60cb62d886f7df20ae5d365cbd3ce13aa007" gracePeriod=30 Oct 30 22:20:06.889108 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:06.889081 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:20:06.890067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:06.890044 4335 patch_prober.go:29] interesting pod/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 22:20:06.890177 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:06.890089 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" podUID="5e3c2933-2586-4001-aa99-c18c1a6d44bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 22:20:07.586403 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:07.586372 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5" Oct 30 22:20:07.642723 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:07.642609 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26693c1f-2409-4b8b-94ed-c52e46123c3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26693c1f-2409-4b8b-94ed-c52e46123c3e-maas-api-xks-ca\") pod \"26693c1f-2409-4b8b-94ed-c52e46123c3e\" (UID: \"26693c1f-2409-4b8b-94ed-c52e46123c3e\") " Oct 30 22:20:07.642892 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:07.642746 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26693c1f-2409-4b8b-94ed-c52e46123c3e-kube-api-access-h6x5q\" (UniqueName: \"kubernetes.io/projected/26693c1f-2409-4b8b-94ed-c52e46123c3e-kube-api-access-h6x5q\") pod \"26693c1f-2409-4b8b-94ed-c52e46123c3e\" (UID: \"26693c1f-2409-4b8b-94ed-c52e46123c3e\") " Oct 30 22:20:07.642892 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:07.642784 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26693c1f-2409-4b8b-94ed-c52e46123c3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26693c1f-2409-4b8b-94ed-c52e46123c3e-maas-api-openshift-ca\") pod \"26693c1f-2409-4b8b-94ed-c52e46123c3e\" (UID: \"26693c1f-2409-4b8b-94ed-c52e46123c3e\") " Oct 30 22:20:07.643201 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:07.643178 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26693c1f-2409-4b8b-94ed-c52e46123c3e-maas-api-openshift-ca" pod "26693c1f-2409-4b8b-94ed-c52e46123c3e" (UID: "26693c1f-2409-4b8b-94ed-c52e46123c3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:07.643337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:07.643318 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26693c1f-2409-4b8b-94ed-c52e46123c3e-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:07.645065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:07.645035 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26693c1f-2409-4b8b-94ed-c52e46123c3e-kube-api-access-h6x5q" pod "26693c1f-2409-4b8b-94ed-c52e46123c3e" (UID: "26693c1f-2409-4b8b-94ed-c52e46123c3e"). InnerVolumeSpecName "kube-api-access-h6x5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:07.645253 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:07.645236 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26693c1f-2409-4b8b-94ed-c52e46123c3e-maas-api-xks-ca" pod "26693c1f-2409-4b8b-94ed-c52e46123c3e" (UID: "26693c1f-2409-4b8b-94ed-c52e46123c3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:20:07.744793 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:07.744758 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6x5q\" (UniqueName: \"kubernetes.io/projected/26693c1f-2409-4b8b-94ed-c52e46123c3e-kube-api-access-h6x5q\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:07.745015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:07.745000 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26693c1f-2409-4b8b-94ed-c52e46123c3e-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:07.888527 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:07.888497 4335 patch_prober.go:29] interesting pod/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 22:20:07.888731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:07.888552 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" podUID="5e3c2933-2586-4001-aa99-c18c1a6d44bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 22:20:08.373565 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:08.373541 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5" Oct 30 22:20:08.373759 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:08.373589 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5" event={"ID":"26693c1f-2409-4b8b-94ed-c52e46123c3e","Type":"ContainerDied","Data":"ddb846a0fd7fa2255e4e112a9259b9e8deb9b36b82256e07b39bf27a0de79166"} Oct 30 22:20:08.373759 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:08.373678 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddb846a0fd7fa2255e4e112a9259b9e8deb9b36b82256e07b39bf27a0de79166" Oct 30 22:20:08.813031 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:08.812680 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:20:08.813031 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:08.812757 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" podUID="63c520ab-1d01-497c-9ac9-a10bfd284e54" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:20:08.887764 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:08.887733 4335 patch_prober.go:29] interesting pod/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 22:20:08.887898 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:08.887795 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" podUID="5e3c2933-2586-4001-aa99-c18c1a6d44bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 22:20:09.644273 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.644239 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh"] Oct 30 22:20:09.645120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.645097 4335 cpu_manager.go:414] "containerMap: removing container" podUID="26693c1f-2409-4b8b-94ed-c52e46123c3e" containerName="revoke-keys" Oct 30 22:20:09.645120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.645118 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26693c1f-2409-4b8b-94ed-c52e46123c3e" containerName="revoke-keys" Oct 30 22:20:09.645275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.645236 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="26693c1f-2409-4b8b-94ed-c52e46123c3e" containerName="revoke-keys" Oct 30 22:20:09.647826 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.647810 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.685966 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.685933 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-workload-socket\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.686097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.685993 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-data\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.686097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.686029 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-credential-socket\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.686097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.686072 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-token\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.686275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.686186 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-envoy\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.686275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.686236 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-podinfo\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.686383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.686277 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zssjx\" (UniqueName: \"kubernetes.io/projected/fc259675-c041-4deb-aaf3-ac3c90c48a05-kube-api-access-zssjx\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.686383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.686306 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-workload-certs\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.686483 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.686384 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc259675-c041-4deb-aaf3-ac3c90c48a05-istiod-ca-cert\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.705668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.703931 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-0f3344-openshift-default-dockercfg-8m9sz\"" Oct 30 22:20:09.762117 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.762086 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh"] Oct 30 22:20:09.790176 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.790144 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-podinfo\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.790991 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.790954 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zssjx\" (UniqueName: \"kubernetes.io/projected/fc259675-c041-4deb-aaf3-ac3c90c48a05-kube-api-access-zssjx\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.791520 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.791469 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-workload-certs\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.792505 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.791919 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-workload-certs\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.792505 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.792440 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc259675-c041-4deb-aaf3-ac3c90c48a05-istiod-ca-cert\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.792730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.791757 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc259675-c041-4deb-aaf3-ac3c90c48a05-istiod-ca-cert\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.792973 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.792953 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-workload-socket\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.793418 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.793397 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-data\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.793575 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.793558 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-credential-socket\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.794044 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.794013 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-token\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.794304 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.794287 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-envoy\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.794494 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.793332 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-workload-socket\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.794653 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.794050 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-credential-socket\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.794757 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.793719 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-podinfo\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.794757 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.793944 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-data\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.837196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.837167 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-envoy\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.852833 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.852800 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zssjx\" (UniqueName: \"kubernetes.io/projected/fc259675-c041-4deb-aaf3-ac3c90c48a05-kube-api-access-zssjx\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.854873 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.854842 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-token\") pod \"e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:09.890633 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.890586 4335 patch_prober.go:29] interesting pod/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 22:20:09.890916 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.890875 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" podUID="5e3c2933-2586-4001-aa99-c18c1a6d44bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 22:20:09.962116 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:09.962037 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:10.429005 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:20:10.428978 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc259675_c041_4deb_aaf3_ac3c90c48a05.slice/crio-a7793369bf0f616f7db9f008162ca71736a2c800b72d0cd75255b7a421b24b77 WatchSource:0}: Error finding container a7793369bf0f616f7db9f008162ca71736a2c800b72d0cd75255b7a421b24b77: Status 404 returned error can't find the container with id a7793369bf0f616f7db9f008162ca71736a2c800b72d0cd75255b7a421b24b77 Oct 30 22:20:10.431498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:10.431473 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh"] Oct 30 22:20:10.431845 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:10.431812 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:20:10.431943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:10.431894 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:20:10.432008 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:10.431939 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:20:10.455675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:10.455646 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" event={"ID":"fc259675-c041-4deb-aaf3-ac3c90c48a05","Type":"ContainerStarted","Data":"a7793369bf0f616f7db9f008162ca71736a2c800b72d0cd75255b7a421b24b77"} Oct 30 22:20:10.887819 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:10.887782 4335 patch_prober.go:29] interesting pod/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 22:20:10.888340 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:10.887840 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" podUID="5e3c2933-2586-4001-aa99-c18c1a6d44bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 22:20:11.405731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:11.405700 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:20:11.465916 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:11.465881 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" event={"ID":"fc259675-c041-4deb-aaf3-ac3c90c48a05","Type":"ContainerStarted","Data":"6e9b65a2165157264b6ca29415de1e43815cc5a204a0cdcdcc1a7ba2f0ed62f8"} Oct 30 22:20:11.590443 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:11.590325 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" podStartSLOduration=2.590297349 podStartE2EDuration="2.590297349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:11.588101791 +0000 UTC m=+2144.688533538" watchObservedRunningTime="2026-10-30 22:20:11.590297349 +0000 UTC m=+2144.690729084" Oct 30 22:20:11.888691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:11.888658 4335 patch_prober.go:29] interesting pod/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 22:20:11.889113 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:11.888716 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" podUID="5e3c2933-2586-4001-aa99-c18c1a6d44bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 22:20:11.963129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:11.963037 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:11.970222 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:11.970117 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:12.156404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.156378 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:20:12.259241 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.259210 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-workload-certs\") pod \"63c520ab-1d01-497c-9ac9-a10bfd284e54\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " Oct 30 22:20:12.259419 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.259276 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c520ab-1d01-497c-9ac9-a10bfd284e54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c520ab-1d01-497c-9ac9-a10bfd284e54-istiod-ca-cert\") pod \"63c520ab-1d01-497c-9ac9-a10bfd284e54\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " Oct 30 22:20:12.259419 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.259297 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-envoy\") pod \"63c520ab-1d01-497c-9ac9-a10bfd284e54\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " Oct 30 22:20:12.259419 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.259331 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-workload-socket\") pod \"63c520ab-1d01-497c-9ac9-a10bfd284e54\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " Oct 30 22:20:12.259657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.259440 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-credential-socket\") pod \"63c520ab-1d01-497c-9ac9-a10bfd284e54\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " Oct 30 22:20:12.259657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.259470 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-token\" (UniqueName: \"kubernetes.io/projected/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-token\") pod \"63c520ab-1d01-497c-9ac9-a10bfd284e54\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " Oct 30 22:20:12.259657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.259499 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c520ab-1d01-497c-9ac9-a10bfd284e54-kube-api-access-j5rfq\" (UniqueName: \"kubernetes.io/projected/63c520ab-1d01-497c-9ac9-a10bfd284e54-kube-api-access-j5rfq\") pod \"63c520ab-1d01-497c-9ac9-a10bfd284e54\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " Oct 30 22:20:12.259657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.259497 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-workload-certs" pod "63c520ab-1d01-497c-9ac9-a10bfd284e54" (UID: "63c520ab-1d01-497c-9ac9-a10bfd284e54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:12.259657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.259588 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-data\") pod \"63c520ab-1d01-497c-9ac9-a10bfd284e54\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " Oct 30 22:20:12.259912 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.259669 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-podinfo\") pod \"63c520ab-1d01-497c-9ac9-a10bfd284e54\" (UID: \"63c520ab-1d01-497c-9ac9-a10bfd284e54\") " Oct 30 22:20:12.259912 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.259711 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-workload-socket" pod "63c520ab-1d01-497c-9ac9-a10bfd284e54" (UID: "63c520ab-1d01-497c-9ac9-a10bfd284e54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:12.259912 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.259787 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c520ab-1d01-497c-9ac9-a10bfd284e54-istiod-ca-cert" pod "63c520ab-1d01-497c-9ac9-a10bfd284e54" (UID: "63c520ab-1d01-497c-9ac9-a10bfd284e54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:12.259912 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.259821 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-credential-socket" pod "63c520ab-1d01-497c-9ac9-a10bfd284e54" (UID: "63c520ab-1d01-497c-9ac9-a10bfd284e54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:12.260225 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.260206 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:12.260288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.260229 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c520ab-1d01-497c-9ac9-a10bfd284e54-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:12.260288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.260241 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:12.260288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.260256 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:12.260522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.260501 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-data" pod "63c520ab-1d01-497c-9ac9-a10bfd284e54" (UID: "63c520ab-1d01-497c-9ac9-a10bfd284e54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:12.261854 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.261827 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-envoy" pod "63c520ab-1d01-497c-9ac9-a10bfd284e54" (UID: "63c520ab-1d01-497c-9ac9-a10bfd284e54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:12.262164 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.262146 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c520ab-1d01-497c-9ac9-a10bfd284e54-kube-api-access-j5rfq" pod "63c520ab-1d01-497c-9ac9-a10bfd284e54" (UID: "63c520ab-1d01-497c-9ac9-a10bfd284e54"). InnerVolumeSpecName "kube-api-access-j5rfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:12.262355 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.262338 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-podinfo" pod "63c520ab-1d01-497c-9ac9-a10bfd284e54" (UID: "63c520ab-1d01-497c-9ac9-a10bfd284e54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:20:12.262422 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.262379 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-token" pod "63c520ab-1d01-497c-9ac9-a10bfd284e54" (UID: "63c520ab-1d01-497c-9ac9-a10bfd284e54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:12.361534 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.361506 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:12.361534 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.361530 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:12.361534 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.361540 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:12.361778 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.361550 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c520ab-1d01-497c-9ac9-a10bfd284e54-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:12.361778 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.361560 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5rfq\" (UniqueName: \"kubernetes.io/projected/63c520ab-1d01-497c-9ac9-a10bfd284e54-kube-api-access-j5rfq\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:12.477199 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.477115 4335 generic.go:356] "Generic (PLEG): container finished" podID="63c520ab-1d01-497c-9ac9-a10bfd284e54" containerID="b9de59cda9a88634dc048d8d522d60cb62d886f7df20ae5d365cbd3ce13aa007" exitCode=0 Oct 30 22:20:12.477199 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.477163 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" Oct 30 22:20:12.477199 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.477171 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" event={"ID":"63c520ab-1d01-497c-9ac9-a10bfd284e54","Type":"ContainerDied","Data":"b9de59cda9a88634dc048d8d522d60cb62d886f7df20ae5d365cbd3ce13aa007"} Oct 30 22:20:12.477452 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.477222 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6" event={"ID":"63c520ab-1d01-497c-9ac9-a10bfd284e54","Type":"ContainerDied","Data":"9d282f1aa4f21345d8c6f459fe924046cbb684bc6cecd00f2aad7631d8783c33"} Oct 30 22:20:12.477452 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.477245 4335 scope.go:122] "RemoveContainer" containerID="b9de59cda9a88634dc048d8d522d60cb62d886f7df20ae5d365cbd3ce13aa007" Oct 30 22:20:12.477774 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.477751 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:12.478921 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.478889 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:20:12.491096 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.491080 4335 scope.go:122] "RemoveContainer" containerID="b9de59cda9a88634dc048d8d522d60cb62d886f7df20ae5d365cbd3ce13aa007" Oct 30 22:20:12.491373 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:20:12.491351 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9de59cda9a88634dc048d8d522d60cb62d886f7df20ae5d365cbd3ce13aa007\": container with ID starting with b9de59cda9a88634dc048d8d522d60cb62d886f7df20ae5d365cbd3ce13aa007 not found: ID does not exist" containerID="b9de59cda9a88634dc048d8d522d60cb62d886f7df20ae5d365cbd3ce13aa007" Oct 30 22:20:12.491438 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.491380 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9de59cda9a88634dc048d8d522d60cb62d886f7df20ae5d365cbd3ce13aa007"} err="failed to get container status \"b9de59cda9a88634dc048d8d522d60cb62d886f7df20ae5d365cbd3ce13aa007\": rpc error: code = NotFound desc = could not find container \"b9de59cda9a88634dc048d8d522d60cb62d886f7df20ae5d365cbd3ce13aa007\": container with ID starting with b9de59cda9a88634dc048d8d522d60cb62d886f7df20ae5d365cbd3ce13aa007 not found: ID does not exist" Oct 30 22:20:12.778984 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.778903 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6"] Oct 30 22:20:12.783679 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.783655 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-gdsj6"] Oct 30 22:20:12.893565 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.893539 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:20:12.894670 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:12.894653 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:20:13.061104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:13.061073 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c520ab-1d01-497c-9ac9-a10bfd284e54" path="/var/lib/kubelet/pods/63c520ab-1d01-497c-9ac9-a10bfd284e54/volumes" Oct 30 22:20:18.439567 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:18.439531 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd"] Oct 30 22:20:18.440533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:18.440512 4335 cpu_manager.go:414] "containerMap: removing container" podUID="63c520ab-1d01-497c-9ac9-a10bfd284e54" containerName="istio-proxy" Oct 30 22:20:18.440533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:18.440532 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c520ab-1d01-497c-9ac9-a10bfd284e54" containerName="istio-proxy" Oct 30 22:20:18.440740 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:18.440708 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c520ab-1d01-497c-9ac9-a10bfd284e54" containerName="istio-proxy" Oct 30 22:20:18.444299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:18.444282 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd" Oct 30 22:20:18.499551 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:18.499523 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5m52\" (UniqueName: \"kubernetes.io/projected/591e6aee-d698-4ced-9c55-0e0db3697bde-kube-api-access-h5m52\") pod \"payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd\" (UID: \"591e6aee-d698-4ced-9c55-0e0db3697bde\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd" Oct 30 22:20:18.499909 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:18.499877 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591e6aee-d698-4ced-9c55-0e0db3697bde-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd\" (UID: \"591e6aee-d698-4ced-9c55-0e0db3697bde\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd" Oct 30 22:20:18.502980 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:18.502956 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd"] Oct 30 22:20:18.546131 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:18.546105 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-0f3344\"" Oct 30 22:20:18.601664 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:18.601628 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591e6aee-d698-4ced-9c55-0e0db3697bde-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd\" (UID: \"591e6aee-d698-4ced-9c55-0e0db3697bde\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd" Oct 30 22:20:18.602037 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:18.602010 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5m52\" (UniqueName: \"kubernetes.io/projected/591e6aee-d698-4ced-9c55-0e0db3697bde-kube-api-access-h5m52\") pod \"payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd\" (UID: \"591e6aee-d698-4ced-9c55-0e0db3697bde\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd" Oct 30 22:20:18.603324 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:18.603283 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591e6aee-d698-4ced-9c55-0e0db3697bde-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd\" (UID: \"591e6aee-d698-4ced-9c55-0e0db3697bde\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd" Oct 30 22:20:18.708534 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:18.708461 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5m52\" (UniqueName: \"kubernetes.io/projected/591e6aee-d698-4ced-9c55-0e0db3697bde-kube-api-access-h5m52\") pod \"payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd\" (UID: \"591e6aee-d698-4ced-9c55-0e0db3697bde\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd" Oct 30 22:20:18.757566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:18.757524 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd" Oct 30 22:20:19.157482 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:20:19.157423 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591e6aee_d698_4ced_9c55_0e0db3697bde.slice/crio-d0d536381d256124a692e8081f7b4f6b5dcf9d5c3f79f04d3ca39b5e5ef72b66 WatchSource:0}: Error finding container d0d536381d256124a692e8081f7b4f6b5dcf9d5c3f79f04d3ca39b5e5ef72b66: Status 404 returned error can't find the container with id d0d536381d256124a692e8081f7b4f6b5dcf9d5c3f79f04d3ca39b5e5ef72b66 Oct 30 22:20:19.158160 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:19.158136 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd"] Oct 30 22:20:19.547094 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:19.547000 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd" event={"ID":"591e6aee-d698-4ced-9c55-0e0db3697bde","Type":"ContainerStarted","Data":"f55da982726289370af5da2b2009e05ed8ea741282f0f761f995111ccce7f454"} Oct 30 22:20:19.547094 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:19.547068 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd" Oct 30 22:20:19.547094 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:19.547085 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd" event={"ID":"591e6aee-d698-4ced-9c55-0e0db3697bde","Type":"ContainerStarted","Data":"d0d536381d256124a692e8081f7b4f6b5dcf9d5c3f79f04d3ca39b5e5ef72b66"} Oct 30 22:20:19.667177 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:19.667127 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd" podStartSLOduration=1.667113188 podStartE2EDuration="1.667113188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:19.664709819 +0000 UTC m=+2152.765141567" watchObservedRunningTime="2026-10-30 22:20:19.667113188 +0000 UTC m=+2152.767544919" Oct 30 22:20:25.323486 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:20:25.323448 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:20:29.331943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:29.331900 4335 scope.go:122] "RemoveContainer" containerID="84603118acc83dd68f48cb41efcee04b7887612a2516670fb0c0eeaae2f39ceb" Oct 30 22:20:30.558561 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:30.558526 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd" Oct 30 22:20:55.325368 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:20:55.325335 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:20:56.053221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:56.053189 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc"] Oct 30 22:20:56.102275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:56.102252 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc9cbb8c-2wmjc"] Oct 30 22:20:57.060629 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:57.060592 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c4a62d-b9e3-49a2-9f30-02197204f81e" path="/var/lib/kubelet/pods/44c4a62d-b9e3-49a2-9f30-02197204f81e/volumes" Oct 30 22:20:57.776389 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:20:57.776359 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:21:01.587174 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.587140 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95"] Oct 30 22:21:01.590651 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.590601 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.644530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.643077 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e3324355-openshift-default-dockercfg-vgcwb\"" Oct 30 22:21:01.664675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.663773 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-workload-socket\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.664675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.663906 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e974fd0-2c92-4626-9673-d202cc596fea-istio-podinfo\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.664675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.663941 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-credential-socket\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.664675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.663969 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e974fd0-2c92-4626-9673-d202cc596fea-istio-token\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.664675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.664021 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-workload-certs\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.664675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.664067 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-istio-envoy\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.664675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.664095 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-istio-data\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.664675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.664152 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e974fd0-2c92-4626-9673-d202cc596fea-istiod-ca-cert\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.664675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.664191 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pffm5\" (UniqueName: \"kubernetes.io/projected/4e974fd0-2c92-4626-9673-d202cc596fea-kube-api-access-pffm5\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.721647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.719682 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95"] Oct 30 22:21:01.780635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.779098 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e974fd0-2c92-4626-9673-d202cc596fea-istio-podinfo\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.780635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.779211 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-credential-socket\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.780635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.779244 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e974fd0-2c92-4626-9673-d202cc596fea-istio-token\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.780635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.779310 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-workload-certs\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.780635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.779364 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-istio-envoy\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.780635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.779400 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-istio-data\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.780635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.779473 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e974fd0-2c92-4626-9673-d202cc596fea-istiod-ca-cert\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.780635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.779517 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pffm5\" (UniqueName: \"kubernetes.io/projected/4e974fd0-2c92-4626-9673-d202cc596fea-kube-api-access-pffm5\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.780635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.779669 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-workload-socket\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.780635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.780295 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-workload-socket\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.790697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.790671 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-credential-socket\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.791521 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.791485 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e974fd0-2c92-4626-9673-d202cc596fea-istiod-ca-cert\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.791775 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.791746 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-istio-data\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.792198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.792177 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-workload-certs\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.853541 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.853464 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e974fd0-2c92-4626-9673-d202cc596fea-istio-token\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.865675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.865646 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e974fd0-2c92-4626-9673-d202cc596fea-istio-podinfo\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.868096 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.868067 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pffm5\" (UniqueName: \"kubernetes.io/projected/4e974fd0-2c92-4626-9673-d202cc596fea-kube-api-access-pffm5\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.874171 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.874146 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-istio-envoy\") pod \"e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:01.930944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:01.930906 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:02.429416 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:02.429378 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95"] Oct 30 22:21:02.462668 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:21:02.462640 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e974fd0_2c92_4626_9673_d202cc596fea.slice/crio-8950b8e3d18c71e47f002ee5b72b601ab78666718d2d7e6b0b18bdeeacd18859 WatchSource:0}: Error finding container 8950b8e3d18c71e47f002ee5b72b601ab78666718d2d7e6b0b18bdeeacd18859: Status 404 returned error can't find the container with id 8950b8e3d18c71e47f002ee5b72b601ab78666718d2d7e6b0b18bdeeacd18859 Oct 30 22:21:02.465393 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:02.465359 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:21:02.465469 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:02.465436 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:21:02.465469 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:02.465463 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:21:02.971558 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:02.971517 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" event={"ID":"4e974fd0-2c92-4626-9673-d202cc596fea","Type":"ContainerStarted","Data":"d9da18cde1e801e291e005dd56c19ad7a9520de55b4d855408e7a44e5f847367"} Oct 30 22:21:02.972017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:02.971608 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" event={"ID":"4e974fd0-2c92-4626-9673-d202cc596fea","Type":"ContainerStarted","Data":"8950b8e3d18c71e47f002ee5b72b601ab78666718d2d7e6b0b18bdeeacd18859"} Oct 30 22:21:03.129506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:03.129441 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" podStartSLOduration=2.129416635 podStartE2EDuration="2.129416635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:03.095480915 +0000 UTC m=+2196.195912649" watchObservedRunningTime="2026-10-30 22:21:03.129416635 +0000 UTC m=+2196.229848369" Oct 30 22:21:03.933124 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:03.933085 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:03.941318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:03.941291 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:03.980326 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:03.980299 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:03.981017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:03.980998 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:21:09.033195 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:09.033158 4335 generic.go:356] "Generic (PLEG): container finished" podID="967921e5-a600-4870-bb43-721aaffbad2d" containerID="73f3917ebf1a46d077acfad490c34bc55de6a39d16003fda80d3eaebdb913436" exitCode=0 Oct 30 22:21:09.033195 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:09.033187 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" event={"ID":"967921e5-a600-4870-bb43-721aaffbad2d","Type":"ContainerDied","Data":"73f3917ebf1a46d077acfad490c34bc55de6a39d16003fda80d3eaebdb913436"} Oct 30 22:21:09.033858 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:09.033666 4335 scope.go:122] "RemoveContainer" containerID="73f3917ebf1a46d077acfad490c34bc55de6a39d16003fda80d3eaebdb913436" Oct 30 22:21:09.035438 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:09.035411 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:21:09.035532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:09.035479 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:21:09.035532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:09.035512 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:21:10.108645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.108057 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" event={"ID":"967921e5-a600-4870-bb43-721aaffbad2d","Type":"ContainerStarted","Data":"7eef06ccfa664791c5427bf9825be891366256539deae56eb0395ad90792255b"} Oct 30 22:21:10.143664 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.143590 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7"] Oct 30 22:21:10.148501 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.148473 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7" Oct 30 22:21:10.208053 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.208018 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e3324355\"" Oct 30 22:21:10.218637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.210820 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e3324355-dockercfg-xrrk2\"" Oct 30 22:21:10.218637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.213279 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:21:10.218637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.213398 4335 patch_prober.go:29] interesting pod/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 22:21:10.218637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.213444 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" podUID="967921e5-a600-4870-bb43-721aaffbad2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 22:21:10.237750 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.237695 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7"] Oct 30 22:21:10.563991 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.563952 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl"] Oct 30 22:21:10.568440 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.568418 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl" Oct 30 22:21:10.569277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.569252 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl"] Oct 30 22:21:10.635566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.635520 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770e0179-e20c-40ab-83f7-b8367aa0afb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7\" (UID: \"770e0179-e20c-40ab-83f7-b8367aa0afb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7" Oct 30 22:21:10.635961 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.635934 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng7gt\" (UniqueName: \"kubernetes.io/projected/770e0179-e20c-40ab-83f7-b8367aa0afb1-kube-api-access-ng7gt\") pod \"payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7\" (UID: \"770e0179-e20c-40ab-83f7-b8367aa0afb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7" Oct 30 22:21:10.739960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.738270 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzd7w\" (UniqueName: \"kubernetes.io/projected/169efb02-ef71-4be0-be80-ca561af3d681-kube-api-access-rzd7w\") pod \"payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl\" (UID: \"169efb02-ef71-4be0-be80-ca561af3d681\") " pod="openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl" Oct 30 22:21:10.739960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.738460 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng7gt\" (UniqueName: \"kubernetes.io/projected/770e0179-e20c-40ab-83f7-b8367aa0afb1-kube-api-access-ng7gt\") pod \"payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7\" (UID: \"770e0179-e20c-40ab-83f7-b8367aa0afb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7" Oct 30 22:21:10.739960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.738965 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169efb02-ef71-4be0-be80-ca561af3d681-plugins-config-volume\") pod \"payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl\" (UID: \"169efb02-ef71-4be0-be80-ca561af3d681\") " pod="openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl" Oct 30 22:21:10.739960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.739037 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770e0179-e20c-40ab-83f7-b8367aa0afb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7\" (UID: \"770e0179-e20c-40ab-83f7-b8367aa0afb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7" Oct 30 22:21:10.740326 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.740079 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770e0179-e20c-40ab-83f7-b8367aa0afb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7\" (UID: \"770e0179-e20c-40ab-83f7-b8367aa0afb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7" Oct 30 22:21:10.823689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.821545 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng7gt\" (UniqueName: \"kubernetes.io/projected/770e0179-e20c-40ab-83f7-b8367aa0afb1-kube-api-access-ng7gt\") pod \"payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7\" (UID: \"770e0179-e20c-40ab-83f7-b8367aa0afb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7" Oct 30 22:21:10.840487 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.840450 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzd7w\" (UniqueName: \"kubernetes.io/projected/169efb02-ef71-4be0-be80-ca561af3d681-kube-api-access-rzd7w\") pod \"payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl\" (UID: \"169efb02-ef71-4be0-be80-ca561af3d681\") " pod="openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl" Oct 30 22:21:10.840677 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.840566 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169efb02-ef71-4be0-be80-ca561af3d681-plugins-config-volume\") pod \"payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl\" (UID: \"169efb02-ef71-4be0-be80-ca561af3d681\") " pod="openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl" Oct 30 22:21:10.842049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.842026 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169efb02-ef71-4be0-be80-ca561af3d681-plugins-config-volume\") pod \"payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl\" (UID: \"169efb02-ef71-4be0-be80-ca561af3d681\") " pod="openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl" Oct 30 22:21:10.930664 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:10.929437 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzd7w\" (UniqueName: \"kubernetes.io/projected/169efb02-ef71-4be0-be80-ca561af3d681-kube-api-access-rzd7w\") pod \"payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl\" (UID: \"169efb02-ef71-4be0-be80-ca561af3d681\") " pod="openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl" Oct 30 22:21:11.062688 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:11.062662 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7" Oct 30 22:21:11.183705 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:11.183479 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl" Oct 30 22:21:11.234827 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:11.234328 4335 patch_prober.go:29] interesting pod/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 22:21:11.234827 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:11.234388 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" podUID="967921e5-a600-4870-bb43-721aaffbad2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 22:21:11.441049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:11.441010 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:21:11.486768 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:21:11.486735 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770e0179_e20c_40ab_83f7_b8367aa0afb1.slice/crio-caab6212cba016a750009dbbe91d05874aed29950e332bed2b218ff23f95f8d8 WatchSource:0}: Error finding container caab6212cba016a750009dbbe91d05874aed29950e332bed2b218ff23f95f8d8: Status 404 returned error can't find the container with id caab6212cba016a750009dbbe91d05874aed29950e332bed2b218ff23f95f8d8 Oct 30 22:21:11.487455 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:11.487431 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7"] Oct 30 22:21:11.672851 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:21:11.672821 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169efb02_ef71_4be0_be80_ca561af3d681.slice/crio-96568b08496ea4eb8d470b63629f47d71a3745e8a34bdc93ee6740954ed30549 WatchSource:0}: Error finding container 96568b08496ea4eb8d470b63629f47d71a3745e8a34bdc93ee6740954ed30549: Status 404 returned error can't find the container with id 96568b08496ea4eb8d470b63629f47d71a3745e8a34bdc93ee6740954ed30549 Oct 30 22:21:11.672973 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:11.672936 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl"] Oct 30 22:21:12.143968 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:12.143935 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl" event={"ID":"169efb02-ef71-4be0-be80-ca561af3d681","Type":"ContainerStarted","Data":"459edf88a0b29de4992f041f53aa6bf172fdf5e3b29bdac7b4195da4e0336c70"} Oct 30 22:21:12.144176 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:12.143992 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl" event={"ID":"169efb02-ef71-4be0-be80-ca561af3d681","Type":"ContainerStarted","Data":"96568b08496ea4eb8d470b63629f47d71a3745e8a34bdc93ee6740954ed30549"} Oct 30 22:21:12.144176 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:12.144057 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl" Oct 30 22:21:12.146093 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:12.146062 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7" event={"ID":"770e0179-e20c-40ab-83f7-b8367aa0afb1","Type":"ContainerStarted","Data":"5e251bf9381cf7eec7b5f05e2a570468c2673393623560121ae83c1d63c54483"} Oct 30 22:21:12.146245 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:12.146102 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7" event={"ID":"770e0179-e20c-40ab-83f7-b8367aa0afb1","Type":"ContainerStarted","Data":"caab6212cba016a750009dbbe91d05874aed29950e332bed2b218ff23f95f8d8"} Oct 30 22:21:12.146245 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:12.146207 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7" Oct 30 22:21:12.225640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:12.224979 4335 patch_prober.go:29] interesting pod/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 22:21:12.225640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:12.225030 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" podUID="967921e5-a600-4870-bb43-721aaffbad2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 22:21:12.261221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:12.261166 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl" podStartSLOduration=2.261149714 podStartE2EDuration="2.261149714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:12.258526552 +0000 UTC m=+2205.358958284" watchObservedRunningTime="2026-10-30 22:21:12.261149714 +0000 UTC m=+2205.361581447" Oct 30 22:21:12.394654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:12.391784 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7" podStartSLOduration=2.391762141 podStartE2EDuration="2.391762141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:12.390713252 +0000 UTC m=+2205.491145058" watchObservedRunningTime="2026-10-30 22:21:12.391762141 +0000 UTC m=+2205.492193874" Oct 30 22:21:13.224640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:13.220761 4335 patch_prober.go:29] interesting pod/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 22:21:13.224640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:13.220831 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" podUID="967921e5-a600-4870-bb43-721aaffbad2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 22:21:14.212640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:14.207923 4335 patch_prober.go:29] interesting pod/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 22:21:14.212640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:14.207993 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" podUID="967921e5-a600-4870-bb43-721aaffbad2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 22:21:15.228045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:15.228008 4335 patch_prober.go:29] interesting pod/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 22:21:15.228666 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:15.228587 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" podUID="967921e5-a600-4870-bb43-721aaffbad2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 22:21:15.508856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:15.508763 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km"] Oct 30 22:21:15.509040 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:15.508964 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" podUID="c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc" containerName="discovery" containerID="cri-o://c434891978a4d097928534d752b6bf1d865a0f21e76963ef3388203cac57b824" gracePeriod=30 Oct 30 22:21:15.895141 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:15.895116 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:21:16.053810 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.053781 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-csr-ca-configmap\") pod \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " Oct 30 22:21:16.053999 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.053833 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-csr-dns-cert\") pod \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " Oct 30 22:21:16.053999 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.053897 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-token\" (UniqueName: \"kubernetes.io/projected/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-token\") pod \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " Oct 30 22:21:16.053999 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.053984 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-local-certs\") pod \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " Oct 30 22:21:16.054187 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.054043 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-kube-api-access-x4sqd\" (UniqueName: \"kubernetes.io/projected/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-kube-api-access-x4sqd\") pod \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " Oct 30 22:21:16.054187 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.054118 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-cacerts\" (UniqueName: \"kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-cacerts\") pod \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " Oct 30 22:21:16.054302 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.054187 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-kubeconfig\") pod \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\" (UID: \"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc\") " Oct 30 22:21:16.054302 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.054257 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-csr-ca-configmap" pod "c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc" (UID: "c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:16.054775 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.054748 4335 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-csr-ca-configmap\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:16.057207 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.057182 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-kube-api-access-x4sqd" pod "c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc" (UID: "c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc"). InnerVolumeSpecName "kube-api-access-x4sqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:16.057316 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.057246 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-local-certs" pod "c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc" (UID: "c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:16.057564 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.057538 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-csr-dns-cert" pod "c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc" (UID: "c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:16.057931 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.057909 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-token" pod "c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc" (UID: "c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:16.059028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.059002 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-cacerts" pod "c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc" (UID: "c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:16.062738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.062715 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-kubeconfig" pod "c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc" (UID: "c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:16.156105 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.156015 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:16.156105 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.156047 4335 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-local-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:16.156105 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.156060 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4sqd\" (UniqueName: \"kubernetes.io/projected/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-kube-api-access-x4sqd\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:16.156105 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.156070 4335 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-cacerts\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:16.156105 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.156079 4335 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-kubeconfig\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:16.156105 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.156088 4335 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc-istio-csr-dns-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:16.205908 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.205872 4335 generic.go:356] "Generic (PLEG): container finished" podID="c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc" containerID="c434891978a4d097928534d752b6bf1d865a0f21e76963ef3388203cac57b824" exitCode=0 Oct 30 22:21:16.206081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.205931 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" Oct 30 22:21:16.206081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.205959 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" event={"ID":"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc","Type":"ContainerDied","Data":"c434891978a4d097928534d752b6bf1d865a0f21e76963ef3388203cac57b824"} Oct 30 22:21:16.206081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.206029 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km" event={"ID":"c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc","Type":"ContainerDied","Data":"2a7d813f453ada802f2c59cea8b322d9e6df0ea7c87d5bddeddc6975448698db"} Oct 30 22:21:16.206081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.206057 4335 scope.go:122] "RemoveContainer" containerID="c434891978a4d097928534d752b6bf1d865a0f21e76963ef3388203cac57b824" Oct 30 22:21:16.219598 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.219576 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:21:16.221013 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.220989 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:21:16.222141 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.222125 4335 scope.go:122] "RemoveContainer" containerID="c434891978a4d097928534d752b6bf1d865a0f21e76963ef3388203cac57b824" Oct 30 22:21:16.222446 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:21:16.222418 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c434891978a4d097928534d752b6bf1d865a0f21e76963ef3388203cac57b824\": container with ID starting with c434891978a4d097928534d752b6bf1d865a0f21e76963ef3388203cac57b824 not found: ID does not exist" containerID="c434891978a4d097928534d752b6bf1d865a0f21e76963ef3388203cac57b824" Oct 30 22:21:16.222536 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.222455 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c434891978a4d097928534d752b6bf1d865a0f21e76963ef3388203cac57b824"} err="failed to get container status \"c434891978a4d097928534d752b6bf1d865a0f21e76963ef3388203cac57b824\": rpc error: code = NotFound desc = could not find container \"c434891978a4d097928534d752b6bf1d865a0f21e76963ef3388203cac57b824\": container with ID starting with c434891978a4d097928534d752b6bf1d865a0f21e76963ef3388203cac57b824 not found: ID does not exist" Oct 30 22:21:16.377251 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.377214 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km"] Oct 30 22:21:16.381631 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:16.381591 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-299km"] Oct 30 22:21:17.060074 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:17.060037 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc" path="/var/lib/kubelet/pods/c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc/volumes" Oct 30 22:21:21.026045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:20.994658 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56"] Oct 30 22:21:21.026045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.006387 4335 cpu_manager.go:414] "containerMap: removing container" podUID="c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc" containerName="discovery" Oct 30 22:21:21.026045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.006412 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc" containerName="discovery" Oct 30 22:21:21.026045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.006770 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f0b281-9719-4ec5-a5d7-2a6a4ad0b5cc" containerName="discovery" Oct 30 22:21:21.092647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.081819 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56" Oct 30 22:21:21.157323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.137757 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh7fv\" (UniqueName: \"kubernetes.io/projected/d2c9a56d-93df-4a25-bec4-6fff59fac92a-kube-api-access-wh7fv\") pod \"payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56\" (UID: \"d2c9a56d-93df-4a25-bec4-6fff59fac92a\") " pod="openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56" Oct 30 22:21:21.157323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.137907 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2c9a56d-93df-4a25-bec4-6fff59fac92a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56\" (UID: \"d2c9a56d-93df-4a25-bec4-6fff59fac92a\") " pod="openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56" Oct 30 22:21:21.172397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.172355 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56"] Oct 30 22:21:21.196542 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.196504 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-67657a-dockercfg-8nbq6\"" Oct 30 22:21:21.196803 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.196703 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-67657a\"" Oct 30 22:21:21.201797 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.201774 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx"] Oct 30 22:21:21.205062 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.205043 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx" Oct 30 22:21:21.239387 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.239349 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b3173ec-26e6-4be5-9d48-911da303107b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx\" (UID: \"0b3173ec-26e6-4be5-9d48-911da303107b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx" Oct 30 22:21:21.239538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.239468 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82bkj\" (UniqueName: \"kubernetes.io/projected/0b3173ec-26e6-4be5-9d48-911da303107b-kube-api-access-82bkj\") pod \"payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx\" (UID: \"0b3173ec-26e6-4be5-9d48-911da303107b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx" Oct 30 22:21:21.239666 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.239542 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2c9a56d-93df-4a25-bec4-6fff59fac92a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56\" (UID: \"d2c9a56d-93df-4a25-bec4-6fff59fac92a\") " pod="openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56" Oct 30 22:21:21.239964 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.239936 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh7fv\" (UniqueName: \"kubernetes.io/projected/d2c9a56d-93df-4a25-bec4-6fff59fac92a-kube-api-access-wh7fv\") pod \"payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56\" (UID: \"d2c9a56d-93df-4a25-bec4-6fff59fac92a\") " pod="openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56" Oct 30 22:21:21.240321 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.240277 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2c9a56d-93df-4a25-bec4-6fff59fac92a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56\" (UID: \"d2c9a56d-93df-4a25-bec4-6fff59fac92a\") " pod="openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56" Oct 30 22:21:21.251032 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.251010 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx"] Oct 30 22:21:21.301136 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.301114 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh7fv\" (UniqueName: \"kubernetes.io/projected/d2c9a56d-93df-4a25-bec4-6fff59fac92a-kube-api-access-wh7fv\") pod \"payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56\" (UID: \"d2c9a56d-93df-4a25-bec4-6fff59fac92a\") " pod="openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56" Oct 30 22:21:21.342545 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.342513 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b3173ec-26e6-4be5-9d48-911da303107b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx\" (UID: \"0b3173ec-26e6-4be5-9d48-911da303107b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx" Oct 30 22:21:21.342716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.342568 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bkj\" (UniqueName: \"kubernetes.io/projected/0b3173ec-26e6-4be5-9d48-911da303107b-kube-api-access-82bkj\") pod \"payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx\" (UID: \"0b3173ec-26e6-4be5-9d48-911da303107b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx" Oct 30 22:21:21.343220 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.343200 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b3173ec-26e6-4be5-9d48-911da303107b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx\" (UID: \"0b3173ec-26e6-4be5-9d48-911da303107b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx" Oct 30 22:21:21.403457 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.403426 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82bkj\" (UniqueName: \"kubernetes.io/projected/0b3173ec-26e6-4be5-9d48-911da303107b-kube-api-access-82bkj\") pod \"payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx\" (UID: \"0b3173ec-26e6-4be5-9d48-911da303107b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx" Oct 30 22:21:21.479710 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.479687 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56" Oct 30 22:21:21.518429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.518407 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx" Oct 30 22:21:21.864442 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.864369 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56"] Oct 30 22:21:21.902128 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:21:21.902099 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c9a56d_93df_4a25_bec4_6fff59fac92a.slice/crio-8766c1cd819f1940a20a513d6c39de02e23975df6cfc565ec05dd68aa0a66ccd WatchSource:0}: Error finding container 8766c1cd819f1940a20a513d6c39de02e23975df6cfc565ec05dd68aa0a66ccd: Status 404 returned error can't find the container with id 8766c1cd819f1940a20a513d6c39de02e23975df6cfc565ec05dd68aa0a66ccd Oct 30 22:21:21.915009 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:21.914985 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx"] Oct 30 22:21:21.917892 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:21:21.917868 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b3173ec_26e6_4be5_9d48_911da303107b.slice/crio-90d05355e912bde1bc7d372dffdbc6d8858498985badb37717a7158b7dee89bb WatchSource:0}: Error finding container 90d05355e912bde1bc7d372dffdbc6d8858498985badb37717a7158b7dee89bb: Status 404 returned error can't find the container with id 90d05355e912bde1bc7d372dffdbc6d8858498985badb37717a7158b7dee89bb Oct 30 22:21:22.260212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:22.260127 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx" event={"ID":"0b3173ec-26e6-4be5-9d48-911da303107b","Type":"ContainerStarted","Data":"fb2bd9c0e515c45ccf7e4509858e4145c41f720232fd2bdc8c2869109a88c8be"} Oct 30 22:21:22.260212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:22.260186 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx" event={"ID":"0b3173ec-26e6-4be5-9d48-911da303107b","Type":"ContainerStarted","Data":"90d05355e912bde1bc7d372dffdbc6d8858498985badb37717a7158b7dee89bb"} Oct 30 22:21:22.260735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:22.260234 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx" Oct 30 22:21:22.261922 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:22.261891 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56" event={"ID":"d2c9a56d-93df-4a25-bec4-6fff59fac92a","Type":"ContainerStarted","Data":"c71c6953b58a3b4625dd3f672eb249693875c14efcd5a497a719574a2246a442"} Oct 30 22:21:22.262069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:22.261932 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56" event={"ID":"d2c9a56d-93df-4a25-bec4-6fff59fac92a","Type":"ContainerStarted","Data":"8766c1cd819f1940a20a513d6c39de02e23975df6cfc565ec05dd68aa0a66ccd"} Oct 30 22:21:22.262069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:22.261971 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56" Oct 30 22:21:22.378479 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:22.378429 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx" podStartSLOduration=1.3784152 podStartE2EDuration="1.3784152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:22.376198392 +0000 UTC m=+2215.476630123" watchObservedRunningTime="2026-10-30 22:21:22.3784152 +0000 UTC m=+2215.478846933" Oct 30 22:21:22.492645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:22.492561 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56" podStartSLOduration=2.492548247 podStartE2EDuration="2.492548247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:22.49127309 +0000 UTC m=+2215.591704823" watchObservedRunningTime="2026-10-30 22:21:22.492548247 +0000 UTC m=+2215.592979978" Oct 30 22:21:23.178041 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:23.178016 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7" Oct 30 22:21:23.178247 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:23.178147 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl" Oct 30 22:21:25.322875 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:21:25.322828 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:21:33.280342 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:33.280309 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56" Oct 30 22:21:33.280849 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:33.280438 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx" Oct 30 22:21:41.693755 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:41.693725 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:21:46.164189 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.162744 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6"] Oct 30 22:21:46.259997 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.245004 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.275849 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.275445 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-data\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.275849 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.275513 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-podinfo\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.275849 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.275699 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istiod-ca-cert\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.275849 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.275756 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-workload-socket\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.276221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.275864 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntb87\" (UniqueName: \"kubernetes.io/projected/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-kube-api-access-ntb87\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.276221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.275910 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-workload-certs\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.276221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.275960 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-credential-socket\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.276221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.276079 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-token\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.276221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.276181 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-envoy\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.277608 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.277576 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6"] Oct 30 22:21:46.321047 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.314019 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-25237a-openshift-default-dockercfg-x9f59\"" Oct 30 22:21:46.377160 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.377133 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-token\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.377346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.377190 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-envoy\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.377346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.377232 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-data\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.377346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.377259 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-podinfo\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.377346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.377333 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istiod-ca-cert\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.377606 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.377381 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-workload-socket\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.377606 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.377422 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntb87\" (UniqueName: \"kubernetes.io/projected/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-kube-api-access-ntb87\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.377606 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.377449 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-workload-certs\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.377606 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.377486 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-credential-socket\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.377855 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.377694 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-data\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.377855 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.377826 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-workload-socket\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.377974 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.377903 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-workload-certs\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.378074 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.378055 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-credential-socket\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.378388 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.378366 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istiod-ca-cert\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.379932 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.379910 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-envoy\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.380309 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.380292 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-podinfo\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.445137 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.443733 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntb87\" (UniqueName: \"kubernetes.io/projected/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-kube-api-access-ntb87\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.445137 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.444305 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-token\") pod \"e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:46.591049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:46.591018 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:47.047547 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:21:47.047516 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58adbdd2_9a58_48f3_bbc2_72c3ac297d5a.slice/crio-59cd99a47e532ed206dd102c8c91e813915dba4867c438a3a6348c1971f8bfea WatchSource:0}: Error finding container 59cd99a47e532ed206dd102c8c91e813915dba4867c438a3a6348c1971f8bfea: Status 404 returned error can't find the container with id 59cd99a47e532ed206dd102c8c91e813915dba4867c438a3a6348c1971f8bfea Oct 30 22:21:47.048768 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:47.048741 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6"] Oct 30 22:21:47.050319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:47.050286 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:21:47.050421 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:47.050353 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:21:47.050421 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:47.050382 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:21:47.561811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:47.561766 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" event={"ID":"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a","Type":"ContainerStarted","Data":"fb0c89f770165718e486203b6306cfe225cf43f9c91266f30e9e15e5dfbbe74f"} Oct 30 22:21:47.562314 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:47.561829 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" event={"ID":"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a","Type":"ContainerStarted","Data":"59cd99a47e532ed206dd102c8c91e813915dba4867c438a3a6348c1971f8bfea"} Oct 30 22:21:47.591526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:47.591495 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:47.789580 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:47.789530 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" podStartSLOduration=2.789515411 podStartE2EDuration="2.789515411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:47.738593763 +0000 UTC m=+2240.839025496" watchObservedRunningTime="2026-10-30 22:21:47.789515411 +0000 UTC m=+2240.889947142" Oct 30 22:21:48.598418 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:48.598385 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:49.585387 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:49.585353 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:49.586238 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:49.586213 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:21:52.613497 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:52.613464 4335 generic.go:356] "Generic (PLEG): container finished" podID="350244ea-9e66-4e83-84ff-ef81766feab1" containerID="1dd72879d2c30c356158d8252ae367488e3cd84f0e38cbadf5946c37f3a434e5" exitCode=0 Oct 30 22:21:52.613894 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:52.613541 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" event={"ID":"350244ea-9e66-4e83-84ff-ef81766feab1","Type":"ContainerDied","Data":"1dd72879d2c30c356158d8252ae367488e3cd84f0e38cbadf5946c37f3a434e5"} Oct 30 22:21:52.613989 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:52.613973 4335 scope.go:122] "RemoveContainer" containerID="1dd72879d2c30c356158d8252ae367488e3cd84f0e38cbadf5946c37f3a434e5" Oct 30 22:21:52.615746 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:52.615721 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:21:52.615807 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:52.615790 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:21:52.615844 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:52.615822 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:21:52.837368 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:52.837337 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:21:53.641235 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:53.641184 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" event={"ID":"350244ea-9e66-4e83-84ff-ef81766feab1","Type":"ContainerStarted","Data":"57e4e534085573f443d0b131e4c68cba76297da253cc80863e78810f4bb72f19"} Oct 30 22:21:54.187858 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:54.187828 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:21:54.189344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:54.189321 4335 patch_prober.go:29] interesting pod/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 22:21:54.189456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:54.189381 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" podUID="350244ea-9e66-4e83-84ff-ef81766feab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 22:21:55.197713 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.188515 4335 patch_prober.go:29] interesting pod/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 22:21:55.197713 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.188577 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" podUID="350244ea-9e66-4e83-84ff-ef81766feab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 22:21:55.260497 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.260465 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7"] Oct 30 22:21:55.267001 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.266977 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7"] Oct 30 22:21:55.267212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.267198 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp"] Oct 30 22:21:55.288635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.288585 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp"] Oct 30 22:21:55.326570 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.304696 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7" Oct 30 22:21:55.333851 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.333818 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp" Oct 30 22:21:55.429640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.426838 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-25237a-dockercfg-v9tww\"" Oct 30 22:21:55.429640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.427500 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-25237a\"" Oct 30 22:21:55.437442 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.436780 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1388034-f774-4af1-bdb7-fa9eb1ec07f5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7\" (UID: \"c1388034-f774-4af1-bdb7-fa9eb1ec07f5\") " pod="openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7" Oct 30 22:21:55.437442 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.436937 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p275l\" (UniqueName: \"kubernetes.io/projected/472bc81d-c1a3-463d-98fe-b98d4576aab7-kube-api-access-p275l\") pod \"payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp\" (UID: \"472bc81d-c1a3-463d-98fe-b98d4576aab7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp" Oct 30 22:21:55.437442 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.437009 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dddr7\" (UniqueName: \"kubernetes.io/projected/c1388034-f774-4af1-bdb7-fa9eb1ec07f5-kube-api-access-dddr7\") pod \"payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7\" (UID: \"c1388034-f774-4af1-bdb7-fa9eb1ec07f5\") " pod="openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7" Oct 30 22:21:55.437442 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.437057 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472bc81d-c1a3-463d-98fe-b98d4576aab7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp\" (UID: \"472bc81d-c1a3-463d-98fe-b98d4576aab7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp" Oct 30 22:21:55.444188 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:21:55.444149 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:21:55.538575 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.538454 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1388034-f774-4af1-bdb7-fa9eb1ec07f5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7\" (UID: \"c1388034-f774-4af1-bdb7-fa9eb1ec07f5\") " pod="openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7" Oct 30 22:21:55.538771 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.538590 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p275l\" (UniqueName: \"kubernetes.io/projected/472bc81d-c1a3-463d-98fe-b98d4576aab7-kube-api-access-p275l\") pod \"payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp\" (UID: \"472bc81d-c1a3-463d-98fe-b98d4576aab7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp" Oct 30 22:21:55.538771 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.538680 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddr7\" (UniqueName: \"kubernetes.io/projected/c1388034-f774-4af1-bdb7-fa9eb1ec07f5-kube-api-access-dddr7\") pod \"payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7\" (UID: \"c1388034-f774-4af1-bdb7-fa9eb1ec07f5\") " pod="openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7" Oct 30 22:21:55.538771 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.538729 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472bc81d-c1a3-463d-98fe-b98d4576aab7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp\" (UID: \"472bc81d-c1a3-463d-98fe-b98d4576aab7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp" Oct 30 22:21:55.539511 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.539489 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472bc81d-c1a3-463d-98fe-b98d4576aab7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp\" (UID: \"472bc81d-c1a3-463d-98fe-b98d4576aab7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp" Oct 30 22:21:55.540382 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.540357 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1388034-f774-4af1-bdb7-fa9eb1ec07f5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7\" (UID: \"c1388034-f774-4af1-bdb7-fa9eb1ec07f5\") " pod="openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7" Oct 30 22:21:55.621637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.621589 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dddr7\" (UniqueName: \"kubernetes.io/projected/c1388034-f774-4af1-bdb7-fa9eb1ec07f5-kube-api-access-dddr7\") pod \"payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7\" (UID: \"c1388034-f774-4af1-bdb7-fa9eb1ec07f5\") " pod="openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7" Oct 30 22:21:55.627350 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.627322 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p275l\" (UniqueName: \"kubernetes.io/projected/472bc81d-c1a3-463d-98fe-b98d4576aab7-kube-api-access-p275l\") pod \"payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp\" (UID: \"472bc81d-c1a3-463d-98fe-b98d4576aab7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp" Oct 30 22:21:55.690989 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.690964 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7" Oct 30 22:21:55.741858 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:55.739075 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp" Oct 30 22:21:56.149319 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:21:56.149287 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472bc81d_c1a3_463d_98fe_b98d4576aab7.slice/crio-486d45ffeb5fcfbb73121a81f89d7e926507ed6134325577c5ba04fc730d3049 WatchSource:0}: Error finding container 486d45ffeb5fcfbb73121a81f89d7e926507ed6134325577c5ba04fc730d3049: Status 404 returned error can't find the container with id 486d45ffeb5fcfbb73121a81f89d7e926507ed6134325577c5ba04fc730d3049 Oct 30 22:21:56.150291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:56.150268 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp"] Oct 30 22:21:56.187537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:56.187513 4335 patch_prober.go:29] interesting pod/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 22:21:56.187671 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:56.187562 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" podUID="350244ea-9e66-4e83-84ff-ef81766feab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 22:21:56.224699 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:56.216366 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7"] Oct 30 22:21:56.737639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:56.736883 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7" event={"ID":"c1388034-f774-4af1-bdb7-fa9eb1ec07f5","Type":"ContainerStarted","Data":"48f8fdc4fd20faf79bfbaec185554586591cc4f6e41c92053422f6d26472da66"} Oct 30 22:21:56.737639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:56.736949 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7" event={"ID":"c1388034-f774-4af1-bdb7-fa9eb1ec07f5","Type":"ContainerStarted","Data":"0f658c45238180561ff75d3afe0b0380b7edb0463eb52fd4485dad77059445a9"} Oct 30 22:21:56.737639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:56.736983 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7" Oct 30 22:21:56.739643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:56.739462 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp" event={"ID":"472bc81d-c1a3-463d-98fe-b98d4576aab7","Type":"ContainerStarted","Data":"c1d02859a0c542c1d3197585f126fdf0b8b12586c52af23eeaf9527fd35c4120"} Oct 30 22:21:56.739643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:56.739522 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp" event={"ID":"472bc81d-c1a3-463d-98fe-b98d4576aab7","Type":"ContainerStarted","Data":"486d45ffeb5fcfbb73121a81f89d7e926507ed6134325577c5ba04fc730d3049"} Oct 30 22:21:56.739643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:56.739602 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp" Oct 30 22:21:56.854464 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:56.853443 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7" podStartSLOduration=2.853426094 podStartE2EDuration="2.853426094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:56.851526781 +0000 UTC m=+2249.951958514" watchObservedRunningTime="2026-10-30 22:21:56.853426094 +0000 UTC m=+2249.953857827" Oct 30 22:21:56.982956 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:56.982369 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp" podStartSLOduration=1.982347182 podStartE2EDuration="1.982347182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:56.974090753 +0000 UTC m=+2250.074522506" watchObservedRunningTime="2026-10-30 22:21:56.982347182 +0000 UTC m=+2250.082778919" Oct 30 22:21:57.188221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:57.188190 4335 patch_prober.go:29] interesting pod/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 22:21:57.188221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:57.188248 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" podUID="350244ea-9e66-4e83-84ff-ef81766feab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 22:21:58.225370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:58.225333 4335 patch_prober.go:29] interesting pod/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 22:21:58.226019 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:58.225985 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" podUID="350244ea-9e66-4e83-84ff-ef81766feab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 22:21:59.193310 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:59.193280 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:21:59.193746 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:59.193722 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:21:59.194460 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:21:59.194442 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:22:07.753086 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:07.753053 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp" Oct 30 22:22:07.753742 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:07.753167 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7" Oct 30 22:22:17.204303 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.204260 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5"] Oct 30 22:22:17.210251 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.210227 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5" Oct 30 22:22:17.252875 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.252854 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgjjj\" (UniqueName: \"kubernetes.io/projected/2c029ed3-d7c8-4a30-8b3b-b60266c70378-kube-api-access-fgjjj\") pod \"maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5\" (UID: \"2c029ed3-d7c8-4a30-8b3b-b60266c70378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5" Oct 30 22:22:17.253002 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.252899 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c029ed3-d7c8-4a30-8b3b-b60266c70378-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5\" (UID: \"2c029ed3-d7c8-4a30-8b3b-b60266c70378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5" Oct 30 22:22:17.253095 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.253073 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c029ed3-d7c8-4a30-8b3b-b60266c70378-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5\" (UID: \"2c029ed3-d7c8-4a30-8b3b-b60266c70378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5" Oct 30 22:22:17.265733 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.265709 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5"] Oct 30 22:22:17.274825 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.274796 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:22:17.274951 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.274796 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:22:17.308577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.308488 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:22:17.308716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.308583 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:22:17.354821 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.354789 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgjjj\" (UniqueName: \"kubernetes.io/projected/2c029ed3-d7c8-4a30-8b3b-b60266c70378-kube-api-access-fgjjj\") pod \"maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5\" (UID: \"2c029ed3-d7c8-4a30-8b3b-b60266c70378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5" Oct 30 22:22:17.354949 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.354844 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c029ed3-d7c8-4a30-8b3b-b60266c70378-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5\" (UID: \"2c029ed3-d7c8-4a30-8b3b-b60266c70378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5" Oct 30 22:22:17.354949 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.354916 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c029ed3-d7c8-4a30-8b3b-b60266c70378-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5\" (UID: \"2c029ed3-d7c8-4a30-8b3b-b60266c70378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5" Oct 30 22:22:17.355652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.355601 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c029ed3-d7c8-4a30-8b3b-b60266c70378-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5\" (UID: \"2c029ed3-d7c8-4a30-8b3b-b60266c70378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5" Oct 30 22:22:17.357488 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.357466 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c029ed3-d7c8-4a30-8b3b-b60266c70378-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5\" (UID: \"2c029ed3-d7c8-4a30-8b3b-b60266c70378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5" Oct 30 22:22:17.416118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.416093 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgjjj\" (UniqueName: \"kubernetes.io/projected/2c029ed3-d7c8-4a30-8b3b-b60266c70378-kube-api-access-fgjjj\") pod \"maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5\" (UID: \"2c029ed3-d7c8-4a30-8b3b-b60266c70378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5" Oct 30 22:22:17.529661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.529574 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5" Oct 30 22:22:17.918083 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:22:17.918053 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c029ed3_d7c8_4a30_8b3b_b60266c70378.slice/crio-2df3660cd90cb20d0eaabd18e84ff325f568a524170c2da8315f6b46c08cc5ad WatchSource:0}: Error finding container 2df3660cd90cb20d0eaabd18e84ff325f568a524170c2da8315f6b46c08cc5ad: Status 404 returned error can't find the container with id 2df3660cd90cb20d0eaabd18e84ff325f568a524170c2da8315f6b46c08cc5ad Oct 30 22:22:17.920448 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.920421 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5"] Oct 30 22:22:17.959415 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:17.959379 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5" event={"ID":"2c029ed3-d7c8-4a30-8b3b-b60266c70378","Type":"ContainerStarted","Data":"2df3660cd90cb20d0eaabd18e84ff325f568a524170c2da8315f6b46c08cc5ad"} Oct 30 22:22:18.970118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:18.970085 4335 generic.go:356] "Generic (PLEG): container finished" podID="2c029ed3-d7c8-4a30-8b3b-b60266c70378" containerID="a94c0abc4963b570dfd55bdc4c7d8e63a8cce5946da8b70f4ea867167ed2bdcb" exitCode=0 Oct 30 22:22:18.970483 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:18.970168 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5" event={"ID":"2c029ed3-d7c8-4a30-8b3b-b60266c70378","Type":"ContainerDied","Data":"a94c0abc4963b570dfd55bdc4c7d8e63a8cce5946da8b70f4ea867167ed2bdcb"} Oct 30 22:22:20.173308 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:20.173281 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5" Oct 30 22:22:20.312963 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:20.312935 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c029ed3-d7c8-4a30-8b3b-b60266c70378-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c029ed3-d7c8-4a30-8b3b-b60266c70378-maas-api-openshift-ca\") pod \"2c029ed3-d7c8-4a30-8b3b-b60266c70378\" (UID: \"2c029ed3-d7c8-4a30-8b3b-b60266c70378\") " Oct 30 22:22:20.313144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:20.313007 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c029ed3-d7c8-4a30-8b3b-b60266c70378-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c029ed3-d7c8-4a30-8b3b-b60266c70378-maas-api-xks-ca\") pod \"2c029ed3-d7c8-4a30-8b3b-b60266c70378\" (UID: \"2c029ed3-d7c8-4a30-8b3b-b60266c70378\") " Oct 30 22:22:20.313259 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:20.313218 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c029ed3-d7c8-4a30-8b3b-b60266c70378-kube-api-access-fgjjj\" (UniqueName: \"kubernetes.io/projected/2c029ed3-d7c8-4a30-8b3b-b60266c70378-kube-api-access-fgjjj\") pod \"2c029ed3-d7c8-4a30-8b3b-b60266c70378\" (UID: \"2c029ed3-d7c8-4a30-8b3b-b60266c70378\") " Oct 30 22:22:20.313338 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:20.313268 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c029ed3-d7c8-4a30-8b3b-b60266c70378-maas-api-openshift-ca" pod "2c029ed3-d7c8-4a30-8b3b-b60266c70378" (UID: "2c029ed3-d7c8-4a30-8b3b-b60266c70378"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:20.313856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:20.313837 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c029ed3-d7c8-4a30-8b3b-b60266c70378-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:20.316777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:20.316752 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c029ed3-d7c8-4a30-8b3b-b60266c70378-kube-api-access-fgjjj" pod "2c029ed3-d7c8-4a30-8b3b-b60266c70378" (UID: "2c029ed3-d7c8-4a30-8b3b-b60266c70378"). InnerVolumeSpecName "kube-api-access-fgjjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:20.316889 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:20.316825 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c029ed3-d7c8-4a30-8b3b-b60266c70378-maas-api-xks-ca" pod "2c029ed3-d7c8-4a30-8b3b-b60266c70378" (UID: "2c029ed3-d7c8-4a30-8b3b-b60266c70378"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:22:20.415599 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:20.415546 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c029ed3-d7c8-4a30-8b3b-b60266c70378-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:20.415599 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:20.415576 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgjjj\" (UniqueName: \"kubernetes.io/projected/2c029ed3-d7c8-4a30-8b3b-b60266c70378-kube-api-access-fgjjj\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:20.988562 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:20.988452 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5" Oct 30 22:22:20.988811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:20.988482 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5" event={"ID":"2c029ed3-d7c8-4a30-8b3b-b60266c70378","Type":"ContainerDied","Data":"2df3660cd90cb20d0eaabd18e84ff325f568a524170c2da8315f6b46c08cc5ad"} Oct 30 22:22:20.988811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:20.988677 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2df3660cd90cb20d0eaabd18e84ff325f568a524170c2da8315f6b46c08cc5ad" Oct 30 22:22:24.047559 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:24.047526 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79"] Oct 30 22:22:24.048108 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:24.047739 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" podUID="c4940530-3248-4c16-b8b0-5959219c8aca" containerName="istio-proxy" containerID="cri-o://5f0d75698a7ae89ff873e0e405186ecd05120de1a01a6c97e436a1668a78236a" gracePeriod=30 Oct 30 22:22:25.321789 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:22:25.321754 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:22:25.628266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:25.628166 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg"] Oct 30 22:22:25.741883 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:25.741850 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj"] Oct 30 22:22:29.321849 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.321783 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:22:29.350134 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.350105 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-credential-socket\") pod \"c4940530-3248-4c16-b8b0-5959219c8aca\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " Oct 30 22:22:29.350272 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.350202 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4940530-3248-4c16-b8b0-5959219c8aca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4940530-3248-4c16-b8b0-5959219c8aca-istio-podinfo\") pod \"c4940530-3248-4c16-b8b0-5959219c8aca\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " Oct 30 22:22:29.350344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.350316 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-istio-envoy\") pod \"c4940530-3248-4c16-b8b0-5959219c8aca\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " Oct 30 22:22:29.350403 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.350348 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-credential-socket" pod "c4940530-3248-4c16-b8b0-5959219c8aca" (UID: "c4940530-3248-4c16-b8b0-5959219c8aca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:29.350403 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.350388 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4940530-3248-4c16-b8b0-5959219c8aca-kube-api-access-gr927\" (UniqueName: \"kubernetes.io/projected/c4940530-3248-4c16-b8b0-5959219c8aca-kube-api-access-gr927\") pod \"c4940530-3248-4c16-b8b0-5959219c8aca\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " Oct 30 22:22:29.350513 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.350466 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4940530-3248-4c16-b8b0-5959219c8aca-istio-token\" (UniqueName: \"kubernetes.io/projected/c4940530-3248-4c16-b8b0-5959219c8aca-istio-token\") pod \"c4940530-3248-4c16-b8b0-5959219c8aca\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " Oct 30 22:22:29.350571 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.350524 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-istio-data\") pod \"c4940530-3248-4c16-b8b0-5959219c8aca\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " Oct 30 22:22:29.350668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.350597 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-workload-socket\") pod \"c4940530-3248-4c16-b8b0-5959219c8aca\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " Oct 30 22:22:29.350736 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.350707 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-workload-certs\") pod \"c4940530-3248-4c16-b8b0-5959219c8aca\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " Oct 30 22:22:29.350793 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.350775 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4940530-3248-4c16-b8b0-5959219c8aca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4940530-3248-4c16-b8b0-5959219c8aca-istiod-ca-cert\") pod \"c4940530-3248-4c16-b8b0-5959219c8aca\" (UID: \"c4940530-3248-4c16-b8b0-5959219c8aca\") " Oct 30 22:22:29.351140 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.351059 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-istio-data" pod "c4940530-3248-4c16-b8b0-5959219c8aca" (UID: "c4940530-3248-4c16-b8b0-5959219c8aca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:29.351285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.351251 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-workload-certs" pod "c4940530-3248-4c16-b8b0-5959219c8aca" (UID: "c4940530-3248-4c16-b8b0-5959219c8aca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:29.351361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.351333 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-workload-socket" pod "c4940530-3248-4c16-b8b0-5959219c8aca" (UID: "c4940530-3248-4c16-b8b0-5959219c8aca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:29.351361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.351349 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4940530-3248-4c16-b8b0-5959219c8aca-istiod-ca-cert" pod "c4940530-3248-4c16-b8b0-5959219c8aca" (UID: "c4940530-3248-4c16-b8b0-5959219c8aca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:29.351536 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.351512 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:29.351607 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.351543 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:29.351607 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.351558 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:29.351607 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.351571 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:29.351607 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.351583 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4940530-3248-4c16-b8b0-5959219c8aca-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:29.352582 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.352556 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4940530-3248-4c16-b8b0-5959219c8aca-istio-podinfo" pod "c4940530-3248-4c16-b8b0-5959219c8aca" (UID: "c4940530-3248-4c16-b8b0-5959219c8aca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:22:29.352814 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.352795 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4940530-3248-4c16-b8b0-5959219c8aca-kube-api-access-gr927" pod "c4940530-3248-4c16-b8b0-5959219c8aca" (UID: "c4940530-3248-4c16-b8b0-5959219c8aca"). InnerVolumeSpecName "kube-api-access-gr927". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:29.353006 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.352990 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-istio-envoy" pod "c4940530-3248-4c16-b8b0-5959219c8aca" (UID: "c4940530-3248-4c16-b8b0-5959219c8aca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:29.353118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.353100 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4940530-3248-4c16-b8b0-5959219c8aca-istio-token" pod "c4940530-3248-4c16-b8b0-5959219c8aca" (UID: "c4940530-3248-4c16-b8b0-5959219c8aca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:29.395607 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.395584 4335 scope.go:122] "RemoveContainer" containerID="468cb04fce24a1e9bec91d55a23923ec30a8469b0e2830d760da01fd16f805e4" Oct 30 22:22:29.406870 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.406846 4335 scope.go:122] "RemoveContainer" containerID="c3a4d1dd92b26f5084f1be32affa39bb24627ad3d59644f6b7c752de50518127" Oct 30 22:22:29.416847 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.416828 4335 scope.go:122] "RemoveContainer" containerID="5f0d75698a7ae89ff873e0e405186ecd05120de1a01a6c97e436a1668a78236a" Oct 30 22:22:29.453168 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.453145 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4940530-3248-4c16-b8b0-5959219c8aca-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:29.453253 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.453170 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4940530-3248-4c16-b8b0-5959219c8aca-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:29.453253 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.453180 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr927\" (UniqueName: \"kubernetes.io/projected/c4940530-3248-4c16-b8b0-5959219c8aca-kube-api-access-gr927\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:29.453253 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:29.453191 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4940530-3248-4c16-b8b0-5959219c8aca-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:30.072823 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:30.072793 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" event={"ID":"c4940530-3248-4c16-b8b0-5959219c8aca","Type":"ContainerDied","Data":"5f0d75698a7ae89ff873e0e405186ecd05120de1a01a6c97e436a1668a78236a"} Oct 30 22:22:30.072977 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:30.072842 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" event={"ID":"c4940530-3248-4c16-b8b0-5959219c8aca","Type":"ContainerDied","Data":"ac845655fd4ac5e0f3a4016c7e5dba0c7fc2429a7b8bce7fe62eb9a89ecdde77"} Oct 30 22:22:30.072977 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:30.072927 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79" Oct 30 22:22:30.316864 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:30.316835 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79"] Oct 30 22:22:30.322278 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:30.322259 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9fbe2-openshift-default-64b8c9665c-4md79"] Oct 30 22:22:30.628681 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:30.628639 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg" podUID="ba8cde58-7add-4dac-97c6-b0ec3482832c" containerName="payload-processing" containerID="cri-o://ebf75bd1e8b4e44dc502eac9bfeeff160fb5420fadbea139dd2e48cf9330e36d" gracePeriod=25 Oct 30 22:22:30.742959 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:30.742931 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj" podUID="b761d5cf-8cdd-42b8-bac9-b6724e06bac5" containerName="payload-pre-processing" containerID="cri-o://fa8bb92d0f1ef2acb646be1926921974ffa99883e6193f8aaf76e53336c370bb" gracePeriod=25 Oct 30 22:22:30.948151 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:30.948121 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg" Oct 30 22:22:31.043723 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.043697 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj" Oct 30 22:22:31.059331 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.059299 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4940530-3248-4c16-b8b0-5959219c8aca" path="/var/lib/kubelet/pods/c4940530-3248-4c16-b8b0-5959219c8aca/volumes" Oct 30 22:22:31.079660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.075632 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba8cde58-7add-4dac-97c6-b0ec3482832c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8cde58-7add-4dac-97c6-b0ec3482832c-plugins-config-volume\") pod \"ba8cde58-7add-4dac-97c6-b0ec3482832c\" (UID: \"ba8cde58-7add-4dac-97c6-b0ec3482832c\") " Oct 30 22:22:31.079660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.075702 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b761d5cf-8cdd-42b8-bac9-b6724e06bac5-kube-api-access-vbjnl\" (UniqueName: \"kubernetes.io/projected/b761d5cf-8cdd-42b8-bac9-b6724e06bac5-kube-api-access-vbjnl\") pod \"b761d5cf-8cdd-42b8-bac9-b6724e06bac5\" (UID: \"b761d5cf-8cdd-42b8-bac9-b6724e06bac5\") " Oct 30 22:22:31.079660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.075817 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba8cde58-7add-4dac-97c6-b0ec3482832c-kube-api-access-cf56h\" (UniqueName: \"kubernetes.io/projected/ba8cde58-7add-4dac-97c6-b0ec3482832c-kube-api-access-cf56h\") pod \"ba8cde58-7add-4dac-97c6-b0ec3482832c\" (UID: \"ba8cde58-7add-4dac-97c6-b0ec3482832c\") " Oct 30 22:22:31.079660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.075974 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba8cde58-7add-4dac-97c6-b0ec3482832c-plugins-config-volume" pod "ba8cde58-7add-4dac-97c6-b0ec3482832c" (UID: "ba8cde58-7add-4dac-97c6-b0ec3482832c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:31.079660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.076037 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b761d5cf-8cdd-42b8-bac9-b6724e06bac5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b761d5cf-8cdd-42b8-bac9-b6724e06bac5-plugins-config-volume\") pod \"b761d5cf-8cdd-42b8-bac9-b6724e06bac5\" (UID: \"b761d5cf-8cdd-42b8-bac9-b6724e06bac5\") " Oct 30 22:22:31.079660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.076423 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b761d5cf-8cdd-42b8-bac9-b6724e06bac5-plugins-config-volume" pod "b761d5cf-8cdd-42b8-bac9-b6724e06bac5" (UID: "b761d5cf-8cdd-42b8-bac9-b6724e06bac5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:31.079660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.076700 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8cde58-7add-4dac-97c6-b0ec3482832c-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:31.079660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.076714 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b761d5cf-8cdd-42b8-bac9-b6724e06bac5-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:31.079660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.078534 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b761d5cf-8cdd-42b8-bac9-b6724e06bac5-kube-api-access-vbjnl" pod "b761d5cf-8cdd-42b8-bac9-b6724e06bac5" (UID: "b761d5cf-8cdd-42b8-bac9-b6724e06bac5"). InnerVolumeSpecName "kube-api-access-vbjnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:31.080198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.079843 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8cde58-7add-4dac-97c6-b0ec3482832c-kube-api-access-cf56h" pod "ba8cde58-7add-4dac-97c6-b0ec3482832c" (UID: "ba8cde58-7add-4dac-97c6-b0ec3482832c"). InnerVolumeSpecName "kube-api-access-cf56h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:31.083860 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.083833 4335 generic.go:356] "Generic (PLEG): container finished" podID="b761d5cf-8cdd-42b8-bac9-b6724e06bac5" containerID="fa8bb92d0f1ef2acb646be1926921974ffa99883e6193f8aaf76e53336c370bb" exitCode=0 Oct 30 22:22:31.083990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.083880 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj" Oct 30 22:22:31.083990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.083917 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj" event={"ID":"b761d5cf-8cdd-42b8-bac9-b6724e06bac5","Type":"ContainerDied","Data":"fa8bb92d0f1ef2acb646be1926921974ffa99883e6193f8aaf76e53336c370bb"} Oct 30 22:22:31.083990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.083968 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj" event={"ID":"b761d5cf-8cdd-42b8-bac9-b6724e06bac5","Type":"ContainerDied","Data":"1e4b1679c7588acd989aa9cf87e76770fe9828dc96a1adf908018467508caf6e"} Oct 30 22:22:31.084183 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.083999 4335 scope.go:122] "RemoveContainer" containerID="fa8bb92d0f1ef2acb646be1926921974ffa99883e6193f8aaf76e53336c370bb" Oct 30 22:22:31.085861 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.085822 4335 generic.go:356] "Generic (PLEG): container finished" podID="ba8cde58-7add-4dac-97c6-b0ec3482832c" containerID="ebf75bd1e8b4e44dc502eac9bfeeff160fb5420fadbea139dd2e48cf9330e36d" exitCode=0 Oct 30 22:22:31.086097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.085902 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg" event={"ID":"ba8cde58-7add-4dac-97c6-b0ec3482832c","Type":"ContainerDied","Data":"ebf75bd1e8b4e44dc502eac9bfeeff160fb5420fadbea139dd2e48cf9330e36d"} Oct 30 22:22:31.086097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.085918 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg" Oct 30 22:22:31.086097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.085962 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg" event={"ID":"ba8cde58-7add-4dac-97c6-b0ec3482832c","Type":"ContainerDied","Data":"a85e3e8ff02f6b0d2591b17687493db5d8d030692d212a4b2f67e54d89f59974"} Oct 30 22:22:31.104833 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.104775 4335 scope.go:122] "RemoveContainer" containerID="fa8bb92d0f1ef2acb646be1926921974ffa99883e6193f8aaf76e53336c370bb" Oct 30 22:22:31.105412 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:22:31.105388 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8bb92d0f1ef2acb646be1926921974ffa99883e6193f8aaf76e53336c370bb\": container with ID starting with fa8bb92d0f1ef2acb646be1926921974ffa99883e6193f8aaf76e53336c370bb not found: ID does not exist" containerID="fa8bb92d0f1ef2acb646be1926921974ffa99883e6193f8aaf76e53336c370bb" Oct 30 22:22:31.105511 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.105422 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8bb92d0f1ef2acb646be1926921974ffa99883e6193f8aaf76e53336c370bb"} err="failed to get container status \"fa8bb92d0f1ef2acb646be1926921974ffa99883e6193f8aaf76e53336c370bb\": rpc error: code = NotFound desc = could not find container \"fa8bb92d0f1ef2acb646be1926921974ffa99883e6193f8aaf76e53336c370bb\": container with ID starting with fa8bb92d0f1ef2acb646be1926921974ffa99883e6193f8aaf76e53336c370bb not found: ID does not exist" Oct 30 22:22:31.105511 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.105447 4335 scope.go:122] "RemoveContainer" containerID="ebf75bd1e8b4e44dc502eac9bfeeff160fb5420fadbea139dd2e48cf9330e36d" Oct 30 22:22:31.117069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.117049 4335 scope.go:122] "RemoveContainer" containerID="ebf75bd1e8b4e44dc502eac9bfeeff160fb5420fadbea139dd2e48cf9330e36d" Oct 30 22:22:31.117368 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:22:31.117349 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf75bd1e8b4e44dc502eac9bfeeff160fb5420fadbea139dd2e48cf9330e36d\": container with ID starting with ebf75bd1e8b4e44dc502eac9bfeeff160fb5420fadbea139dd2e48cf9330e36d not found: ID does not exist" containerID="ebf75bd1e8b4e44dc502eac9bfeeff160fb5420fadbea139dd2e48cf9330e36d" Oct 30 22:22:31.117448 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.117376 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf75bd1e8b4e44dc502eac9bfeeff160fb5420fadbea139dd2e48cf9330e36d"} err="failed to get container status \"ebf75bd1e8b4e44dc502eac9bfeeff160fb5420fadbea139dd2e48cf9330e36d\": rpc error: code = NotFound desc = could not find container \"ebf75bd1e8b4e44dc502eac9bfeeff160fb5420fadbea139dd2e48cf9330e36d\": container with ID starting with ebf75bd1e8b4e44dc502eac9bfeeff160fb5420fadbea139dd2e48cf9330e36d not found: ID does not exist" Oct 30 22:22:31.178149 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.178092 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbjnl\" (UniqueName: \"kubernetes.io/projected/b761d5cf-8cdd-42b8-bac9-b6724e06bac5-kube-api-access-vbjnl\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:31.178149 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.178115 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf56h\" (UniqueName: \"kubernetes.io/projected/ba8cde58-7add-4dac-97c6-b0ec3482832c-kube-api-access-cf56h\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:31.301204 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.301176 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg"] Oct 30 22:22:31.305695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.305673 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bdc5fe-76694b6445-dckrg"] Oct 30 22:22:31.415046 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.415021 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj"] Oct 30 22:22:31.421555 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:31.421531 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bdc5fe-87f585d75-pd6wj"] Oct 30 22:22:33.061204 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:33.061168 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b761d5cf-8cdd-42b8-bac9-b6724e06bac5" path="/var/lib/kubelet/pods/b761d5cf-8cdd-42b8-bac9-b6724e06bac5/volumes" Oct 30 22:22:33.061789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:33.061768 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba8cde58-7add-4dac-97c6-b0ec3482832c" path="/var/lib/kubelet/pods/ba8cde58-7add-4dac-97c6-b0ec3482832c/volumes" Oct 30 22:22:43.251946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.251870 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4"] Oct 30 22:22:43.252651 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.252636 4335 cpu_manager.go:414] "containerMap: removing container" podUID="2c029ed3-d7c8-4a30-8b3b-b60266c70378" containerName="revoke-keys" Oct 30 22:22:43.252718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.252651 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c029ed3-d7c8-4a30-8b3b-b60266c70378" containerName="revoke-keys" Oct 30 22:22:43.252718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.252680 4335 cpu_manager.go:414] "containerMap: removing container" podUID="ba8cde58-7add-4dac-97c6-b0ec3482832c" containerName="payload-processing" Oct 30 22:22:43.252718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.252687 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba8cde58-7add-4dac-97c6-b0ec3482832c" containerName="payload-processing" Oct 30 22:22:43.252718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.252708 4335 cpu_manager.go:414] "containerMap: removing container" podUID="c4940530-3248-4c16-b8b0-5959219c8aca" containerName="istio-proxy" Oct 30 22:22:43.252718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.252713 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4940530-3248-4c16-b8b0-5959219c8aca" containerName="istio-proxy" Oct 30 22:22:43.252878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.252726 4335 cpu_manager.go:414] "containerMap: removing container" podUID="b761d5cf-8cdd-42b8-bac9-b6724e06bac5" containerName="payload-pre-processing" Oct 30 22:22:43.252878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.252731 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b761d5cf-8cdd-42b8-bac9-b6724e06bac5" containerName="payload-pre-processing" Oct 30 22:22:43.252878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.252843 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c029ed3-d7c8-4a30-8b3b-b60266c70378" containerName="revoke-keys" Oct 30 22:22:43.252878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.252853 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba8cde58-7add-4dac-97c6-b0ec3482832c" containerName="payload-processing" Oct 30 22:22:43.252878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.252862 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4940530-3248-4c16-b8b0-5959219c8aca" containerName="istio-proxy" Oct 30 22:22:43.252878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.252870 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="b761d5cf-8cdd-42b8-bac9-b6724e06bac5" containerName="payload-pre-processing" Oct 30 22:22:43.258130 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.258111 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4" Oct 30 22:22:43.316233 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.316209 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4"] Oct 30 22:22:43.359896 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.359783 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:22:43.359896 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.359815 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:22:43.359896 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.359835 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:22:43.359896 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.359849 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:22:43.426837 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.426804 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b2019c-2364-4717-a710-b91b11ea192b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4\" (UID: \"10b2019c-2364-4717-a710-b91b11ea192b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4" Oct 30 22:22:43.426989 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.426954 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp8f6\" (UniqueName: \"kubernetes.io/projected/10b2019c-2364-4717-a710-b91b11ea192b-kube-api-access-mp8f6\") pod \"maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4\" (UID: \"10b2019c-2364-4717-a710-b91b11ea192b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4" Oct 30 22:22:43.427057 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.427013 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b2019c-2364-4717-a710-b91b11ea192b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4\" (UID: \"10b2019c-2364-4717-a710-b91b11ea192b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4" Oct 30 22:22:43.528247 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.528174 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b2019c-2364-4717-a710-b91b11ea192b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4\" (UID: \"10b2019c-2364-4717-a710-b91b11ea192b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4" Oct 30 22:22:43.528410 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.528329 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b2019c-2364-4717-a710-b91b11ea192b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4\" (UID: \"10b2019c-2364-4717-a710-b91b11ea192b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4" Oct 30 22:22:43.528485 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.528463 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp8f6\" (UniqueName: \"kubernetes.io/projected/10b2019c-2364-4717-a710-b91b11ea192b-kube-api-access-mp8f6\") pod \"maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4\" (UID: \"10b2019c-2364-4717-a710-b91b11ea192b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4" Oct 30 22:22:43.528973 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.528952 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b2019c-2364-4717-a710-b91b11ea192b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4\" (UID: \"10b2019c-2364-4717-a710-b91b11ea192b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4" Oct 30 22:22:43.531171 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.531142 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b2019c-2364-4717-a710-b91b11ea192b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4\" (UID: \"10b2019c-2364-4717-a710-b91b11ea192b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4" Oct 30 22:22:43.589654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.589602 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp8f6\" (UniqueName: \"kubernetes.io/projected/10b2019c-2364-4717-a710-b91b11ea192b-kube-api-access-mp8f6\") pod \"maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4\" (UID: \"10b2019c-2364-4717-a710-b91b11ea192b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4" Oct 30 22:22:43.872364 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:43.872338 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4" Oct 30 22:22:44.265355 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:22:44.265326 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b2019c_2364_4717_a710_b91b11ea192b.slice/crio-19bda4d8e1e55e5ab12cc502a811d1ecbf461471fa111b128b44e59e82de75bc WatchSource:0}: Error finding container 19bda4d8e1e55e5ab12cc502a811d1ecbf461471fa111b128b44e59e82de75bc: Status 404 returned error can't find the container with id 19bda4d8e1e55e5ab12cc502a811d1ecbf461471fa111b128b44e59e82de75bc Oct 30 22:22:44.265817 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:44.265413 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4"] Oct 30 22:22:45.218662 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:45.218628 4335 generic.go:356] "Generic (PLEG): container finished" podID="10b2019c-2364-4717-a710-b91b11ea192b" containerID="8b3c45f84291e0d9a59c7e4dd5e475000c6903c746dd7db8e424ae69524f59a7" exitCode=0 Oct 30 22:22:45.218792 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:45.218663 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4" event={"ID":"10b2019c-2364-4717-a710-b91b11ea192b","Type":"ContainerDied","Data":"8b3c45f84291e0d9a59c7e4dd5e475000c6903c746dd7db8e424ae69524f59a7"} Oct 30 22:22:45.218792 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:45.218726 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4" event={"ID":"10b2019c-2364-4717-a710-b91b11ea192b","Type":"ContainerStarted","Data":"19bda4d8e1e55e5ab12cc502a811d1ecbf461471fa111b128b44e59e82de75bc"} Oct 30 22:22:46.411274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:46.411247 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4" Oct 30 22:22:46.468830 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:46.468800 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10b2019c-2364-4717-a710-b91b11ea192b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b2019c-2364-4717-a710-b91b11ea192b-maas-api-xks-ca\") pod \"10b2019c-2364-4717-a710-b91b11ea192b\" (UID: \"10b2019c-2364-4717-a710-b91b11ea192b\") " Oct 30 22:22:46.468989 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:46.468863 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b2019c-2364-4717-a710-b91b11ea192b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b2019c-2364-4717-a710-b91b11ea192b-maas-api-openshift-ca\") pod \"10b2019c-2364-4717-a710-b91b11ea192b\" (UID: \"10b2019c-2364-4717-a710-b91b11ea192b\") " Oct 30 22:22:46.468989 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:46.468949 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b2019c-2364-4717-a710-b91b11ea192b-kube-api-access-mp8f6\" (UniqueName: \"kubernetes.io/projected/10b2019c-2364-4717-a710-b91b11ea192b-kube-api-access-mp8f6\") pod \"10b2019c-2364-4717-a710-b91b11ea192b\" (UID: \"10b2019c-2364-4717-a710-b91b11ea192b\") " Oct 30 22:22:46.469282 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:46.469257 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b2019c-2364-4717-a710-b91b11ea192b-maas-api-openshift-ca" pod "10b2019c-2364-4717-a710-b91b11ea192b" (UID: "10b2019c-2364-4717-a710-b91b11ea192b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:46.469415 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:46.469399 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b2019c-2364-4717-a710-b91b11ea192b-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:46.471087 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:46.471059 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b2019c-2364-4717-a710-b91b11ea192b-kube-api-access-mp8f6" pod "10b2019c-2364-4717-a710-b91b11ea192b" (UID: "10b2019c-2364-4717-a710-b91b11ea192b"). InnerVolumeSpecName "kube-api-access-mp8f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:46.471183 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:46.471141 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10b2019c-2364-4717-a710-b91b11ea192b-maas-api-xks-ca" pod "10b2019c-2364-4717-a710-b91b11ea192b" (UID: "10b2019c-2364-4717-a710-b91b11ea192b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:22:46.570779 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:46.570746 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp8f6\" (UniqueName: \"kubernetes.io/projected/10b2019c-2364-4717-a710-b91b11ea192b-kube-api-access-mp8f6\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:46.570779 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:46.570774 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b2019c-2364-4717-a710-b91b11ea192b-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:47.238573 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:47.238544 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4" Oct 30 22:22:47.238798 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:47.238586 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4" event={"ID":"10b2019c-2364-4717-a710-b91b11ea192b","Type":"ContainerDied","Data":"19bda4d8e1e55e5ab12cc502a811d1ecbf461471fa111b128b44e59e82de75bc"} Oct 30 22:22:47.238798 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:47.238655 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19bda4d8e1e55e5ab12cc502a811d1ecbf461471fa111b128b44e59e82de75bc" Oct 30 22:22:49.907537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:49.907500 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:22:52.057463 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:52.057410 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d"] Oct 30 22:22:52.106169 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:52.106136 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d8626ba-rlf9d"] Oct 30 22:22:53.059928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:53.059891 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0002b46b-ce78-46cd-847a-6b9c334715fd" path="/var/lib/kubelet/pods/0002b46b-ce78-46cd-847a-6b9c334715fd/volumes" Oct 30 22:22:55.318356 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:55.318283 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn"] Oct 30 22:22:55.327354 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:22:55.327322 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:22:55.375196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:55.375160 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs"] Oct 30 22:22:58.563434 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.563397 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s"] Oct 30 22:22:58.564223 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.564205 4335 cpu_manager.go:414] "containerMap: removing container" podUID="10b2019c-2364-4717-a710-b91b11ea192b" containerName="revoke-keys" Oct 30 22:22:58.564223 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.564222 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b2019c-2364-4717-a710-b91b11ea192b" containerName="revoke-keys" Oct 30 22:22:58.564361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.564339 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b2019c-2364-4717-a710-b91b11ea192b" containerName="revoke-keys" Oct 30 22:22:58.568452 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.568428 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s" Oct 30 22:22:58.640854 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.640820 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s"] Oct 30 22:22:58.641756 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.641723 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6dc8\" (UniqueName: \"kubernetes.io/projected/4229ef1e-4d03-4100-88d9-f42cd214561e-kube-api-access-q6dc8\") pod \"maas-api-revoke-keys-e2e-inf-a-a78655-5776s\" (UID: \"4229ef1e-4d03-4100-88d9-f42cd214561e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s" Oct 30 22:22:58.642021 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.641999 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4229ef1e-4d03-4100-88d9-f42cd214561e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a78655-5776s\" (UID: \"4229ef1e-4d03-4100-88d9-f42cd214561e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s" Oct 30 22:22:58.642101 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.642057 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4229ef1e-4d03-4100-88d9-f42cd214561e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a78655-5776s\" (UID: \"4229ef1e-4d03-4100-88d9-f42cd214561e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s" Oct 30 22:22:58.682265 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.682236 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:22:58.682489 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.682320 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:22:58.682489 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.682374 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:22:58.682489 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.682398 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:22:58.743411 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.743388 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4229ef1e-4d03-4100-88d9-f42cd214561e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a78655-5776s\" (UID: \"4229ef1e-4d03-4100-88d9-f42cd214561e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s" Oct 30 22:22:58.743567 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.743426 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4229ef1e-4d03-4100-88d9-f42cd214561e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a78655-5776s\" (UID: \"4229ef1e-4d03-4100-88d9-f42cd214561e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s" Oct 30 22:22:58.743567 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.743494 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6dc8\" (UniqueName: \"kubernetes.io/projected/4229ef1e-4d03-4100-88d9-f42cd214561e-kube-api-access-q6dc8\") pod \"maas-api-revoke-keys-e2e-inf-a-a78655-5776s\" (UID: \"4229ef1e-4d03-4100-88d9-f42cd214561e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s" Oct 30 22:22:58.744166 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.744143 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4229ef1e-4d03-4100-88d9-f42cd214561e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a78655-5776s\" (UID: \"4229ef1e-4d03-4100-88d9-f42cd214561e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s" Oct 30 22:22:58.746008 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.745985 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4229ef1e-4d03-4100-88d9-f42cd214561e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a78655-5776s\" (UID: \"4229ef1e-4d03-4100-88d9-f42cd214561e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s" Oct 30 22:22:58.805834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.805801 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6dc8\" (UniqueName: \"kubernetes.io/projected/4229ef1e-4d03-4100-88d9-f42cd214561e-kube-api-access-q6dc8\") pod \"maas-api-revoke-keys-e2e-inf-a-a78655-5776s\" (UID: \"4229ef1e-4d03-4100-88d9-f42cd214561e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s" Oct 30 22:22:58.883004 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:58.882939 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s" Oct 30 22:22:59.275867 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:59.275839 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s"] Oct 30 22:22:59.276822 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:22:59.276781 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4229ef1e_4d03_4100_88d9_f42cd214561e.slice/crio-fb5ff6f0a4fe8eb7317999940e106789a593eb1e70eb6c45fdfef318a11dbdaf WatchSource:0}: Error finding container fb5ff6f0a4fe8eb7317999940e106789a593eb1e70eb6c45fdfef318a11dbdaf: Status 404 returned error can't find the container with id fb5ff6f0a4fe8eb7317999940e106789a593eb1e70eb6c45fdfef318a11dbdaf Oct 30 22:22:59.355803 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:59.355773 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s" event={"ID":"4229ef1e-4d03-4100-88d9-f42cd214561e","Type":"ContainerStarted","Data":"fb5ff6f0a4fe8eb7317999940e106789a593eb1e70eb6c45fdfef318a11dbdaf"} Oct 30 22:22:59.683383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:59.683353 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4"] Oct 30 22:22:59.688265 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:22:59.688237 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2d3647-6bzd4"] Oct 30 22:23:00.318968 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:00.318919 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn" podUID="7d13516d-0abb-4663-a016-45a93612d102" containerName="payload-processing" containerID="cri-o://1e6a5d668425a9206f48bb960101ef328d87974b3f8c169fa63484ad8b0f6c58" gracePeriod=25 Oct 30 22:23:00.368595 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:00.368563 4335 generic.go:356] "Generic (PLEG): container finished" podID="4229ef1e-4d03-4100-88d9-f42cd214561e" containerID="e19633ccc8f4f5136721bfe2209f143841e1940be5e332154e457975de669a40" exitCode=0 Oct 30 22:23:00.368742 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:00.368675 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s" event={"ID":"4229ef1e-4d03-4100-88d9-f42cd214561e","Type":"ContainerDied","Data":"e19633ccc8f4f5136721bfe2209f143841e1940be5e332154e457975de669a40"} Oct 30 22:23:00.375668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:00.375646 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs" podUID="dbecb7f4-f382-4797-aaca-f4c362171cd9" containerName="payload-pre-processing" containerID="cri-o://4b5658a5b131b2a598b41a6c17dd2a07e01ad35bda6b9fbf986569c35e7e94ab" gracePeriod=25 Oct 30 22:23:00.679272 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:00.679229 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn" Oct 30 22:23:00.703695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:00.703665 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs" Oct 30 22:23:00.782871 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:00.782845 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d13516d-0abb-4663-a016-45a93612d102-kube-api-access-69txr\" (UniqueName: \"kubernetes.io/projected/7d13516d-0abb-4663-a016-45a93612d102-kube-api-access-69txr\") pod \"7d13516d-0abb-4663-a016-45a93612d102\" (UID: \"7d13516d-0abb-4663-a016-45a93612d102\") " Oct 30 22:23:00.783000 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:00.782932 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d13516d-0abb-4663-a016-45a93612d102-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d13516d-0abb-4663-a016-45a93612d102-plugins-config-volume\") pod \"7d13516d-0abb-4663-a016-45a93612d102\" (UID: \"7d13516d-0abb-4663-a016-45a93612d102\") " Oct 30 22:23:00.783053 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:00.783002 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbecb7f4-f382-4797-aaca-f4c362171cd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbecb7f4-f382-4797-aaca-f4c362171cd9-plugins-config-volume\") pod \"dbecb7f4-f382-4797-aaca-f4c362171cd9\" (UID: \"dbecb7f4-f382-4797-aaca-f4c362171cd9\") " Oct 30 22:23:00.783101 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:00.783054 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbecb7f4-f382-4797-aaca-f4c362171cd9-kube-api-access-qmrzd\" (UniqueName: \"kubernetes.io/projected/dbecb7f4-f382-4797-aaca-f4c362171cd9-kube-api-access-qmrzd\") pod \"dbecb7f4-f382-4797-aaca-f4c362171cd9\" (UID: \"dbecb7f4-f382-4797-aaca-f4c362171cd9\") " Oct 30 22:23:00.783396 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:00.783372 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbecb7f4-f382-4797-aaca-f4c362171cd9-plugins-config-volume" pod "dbecb7f4-f382-4797-aaca-f4c362171cd9" (UID: "dbecb7f4-f382-4797-aaca-f4c362171cd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:00.783488 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:00.783370 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d13516d-0abb-4663-a016-45a93612d102-plugins-config-volume" pod "7d13516d-0abb-4663-a016-45a93612d102" (UID: "7d13516d-0abb-4663-a016-45a93612d102"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:00.783798 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:00.783781 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d13516d-0abb-4663-a016-45a93612d102-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:00.783854 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:00.783801 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbecb7f4-f382-4797-aaca-f4c362171cd9-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:00.785230 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:00.785202 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbecb7f4-f382-4797-aaca-f4c362171cd9-kube-api-access-qmrzd" pod "dbecb7f4-f382-4797-aaca-f4c362171cd9" (UID: "dbecb7f4-f382-4797-aaca-f4c362171cd9"). InnerVolumeSpecName "kube-api-access-qmrzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:00.785352 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:00.785330 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d13516d-0abb-4663-a016-45a93612d102-kube-api-access-69txr" pod "7d13516d-0abb-4663-a016-45a93612d102" (UID: "7d13516d-0abb-4663-a016-45a93612d102"). InnerVolumeSpecName "kube-api-access-69txr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:00.885775 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:00.885700 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69txr\" (UniqueName: \"kubernetes.io/projected/7d13516d-0abb-4663-a016-45a93612d102-kube-api-access-69txr\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:00.885775 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:00.885731 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmrzd\" (UniqueName: \"kubernetes.io/projected/dbecb7f4-f382-4797-aaca-f4c362171cd9-kube-api-access-qmrzd\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:01.060792 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.060756 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b2019c-2364-4717-a710-b91b11ea192b" path="/var/lib/kubelet/pods/10b2019c-2364-4717-a710-b91b11ea192b/volumes" Oct 30 22:23:01.379279 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.379242 4335 generic.go:356] "Generic (PLEG): container finished" podID="dbecb7f4-f382-4797-aaca-f4c362171cd9" containerID="4b5658a5b131b2a598b41a6c17dd2a07e01ad35bda6b9fbf986569c35e7e94ab" exitCode=0 Oct 30 22:23:01.379528 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.379294 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs" Oct 30 22:23:01.379528 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.379333 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs" event={"ID":"dbecb7f4-f382-4797-aaca-f4c362171cd9","Type":"ContainerDied","Data":"4b5658a5b131b2a598b41a6c17dd2a07e01ad35bda6b9fbf986569c35e7e94ab"} Oct 30 22:23:01.379528 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.379403 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs" event={"ID":"dbecb7f4-f382-4797-aaca-f4c362171cd9","Type":"ContainerDied","Data":"c24c7959c2a4f67ccd4c259420e00075839184386ae958a46780e2999dc6a8b2"} Oct 30 22:23:01.379528 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.379443 4335 scope.go:122] "RemoveContainer" containerID="4b5658a5b131b2a598b41a6c17dd2a07e01ad35bda6b9fbf986569c35e7e94ab" Oct 30 22:23:01.381088 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.381067 4335 generic.go:356] "Generic (PLEG): container finished" podID="7d13516d-0abb-4663-a016-45a93612d102" containerID="1e6a5d668425a9206f48bb960101ef328d87974b3f8c169fa63484ad8b0f6c58" exitCode=0 Oct 30 22:23:01.381193 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.381128 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn" Oct 30 22:23:01.381260 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.381215 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn" event={"ID":"7d13516d-0abb-4663-a016-45a93612d102","Type":"ContainerDied","Data":"1e6a5d668425a9206f48bb960101ef328d87974b3f8c169fa63484ad8b0f6c58"} Oct 30 22:23:01.381322 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.381256 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn" event={"ID":"7d13516d-0abb-4663-a016-45a93612d102","Type":"ContainerDied","Data":"bb3730cc629081595847b366810b31bebd2b692f42b50a07cb7f224a2430bbdc"} Oct 30 22:23:01.393822 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.393805 4335 scope.go:122] "RemoveContainer" containerID="4b5658a5b131b2a598b41a6c17dd2a07e01ad35bda6b9fbf986569c35e7e94ab" Oct 30 22:23:01.394148 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:23:01.394129 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b5658a5b131b2a598b41a6c17dd2a07e01ad35bda6b9fbf986569c35e7e94ab\": container with ID starting with 4b5658a5b131b2a598b41a6c17dd2a07e01ad35bda6b9fbf986569c35e7e94ab not found: ID does not exist" containerID="4b5658a5b131b2a598b41a6c17dd2a07e01ad35bda6b9fbf986569c35e7e94ab" Oct 30 22:23:01.394211 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.394155 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5658a5b131b2a598b41a6c17dd2a07e01ad35bda6b9fbf986569c35e7e94ab"} err="failed to get container status \"4b5658a5b131b2a598b41a6c17dd2a07e01ad35bda6b9fbf986569c35e7e94ab\": rpc error: code = NotFound desc = could not find container \"4b5658a5b131b2a598b41a6c17dd2a07e01ad35bda6b9fbf986569c35e7e94ab\": container with ID starting with 4b5658a5b131b2a598b41a6c17dd2a07e01ad35bda6b9fbf986569c35e7e94ab not found: ID does not exist" Oct 30 22:23:01.394211 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.394174 4335 scope.go:122] "RemoveContainer" containerID="1e6a5d668425a9206f48bb960101ef328d87974b3f8c169fa63484ad8b0f6c58" Oct 30 22:23:01.411495 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.411473 4335 scope.go:122] "RemoveContainer" containerID="1e6a5d668425a9206f48bb960101ef328d87974b3f8c169fa63484ad8b0f6c58" Oct 30 22:23:01.411946 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:23:01.411923 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6a5d668425a9206f48bb960101ef328d87974b3f8c169fa63484ad8b0f6c58\": container with ID starting with 1e6a5d668425a9206f48bb960101ef328d87974b3f8c169fa63484ad8b0f6c58 not found: ID does not exist" containerID="1e6a5d668425a9206f48bb960101ef328d87974b3f8c169fa63484ad8b0f6c58" Oct 30 22:23:01.412048 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.411962 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6a5d668425a9206f48bb960101ef328d87974b3f8c169fa63484ad8b0f6c58"} err="failed to get container status \"1e6a5d668425a9206f48bb960101ef328d87974b3f8c169fa63484ad8b0f6c58\": rpc error: code = NotFound desc = could not find container \"1e6a5d668425a9206f48bb960101ef328d87974b3f8c169fa63484ad8b0f6c58\": container with ID starting with 1e6a5d668425a9206f48bb960101ef328d87974b3f8c169fa63484ad8b0f6c58 not found: ID does not exist" Oct 30 22:23:01.548560 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.548526 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn"] Oct 30 22:23:01.553461 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.553442 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2d3647-77d9c9dcd5-vfcvn"] Oct 30 22:23:01.576532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.576504 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s" Oct 30 22:23:01.661884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.661806 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs"] Oct 30 22:23:01.666283 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.666263 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2d3647-ccbb886b7-hllfs"] Oct 30 22:23:01.697491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.697461 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4229ef1e-4d03-4100-88d9-f42cd214561e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4229ef1e-4d03-4100-88d9-f42cd214561e-maas-api-openshift-ca\") pod \"4229ef1e-4d03-4100-88d9-f42cd214561e\" (UID: \"4229ef1e-4d03-4100-88d9-f42cd214561e\") " Oct 30 22:23:01.697639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.697609 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4229ef1e-4d03-4100-88d9-f42cd214561e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4229ef1e-4d03-4100-88d9-f42cd214561e-maas-api-xks-ca\") pod \"4229ef1e-4d03-4100-88d9-f42cd214561e\" (UID: \"4229ef1e-4d03-4100-88d9-f42cd214561e\") " Oct 30 22:23:01.697720 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.697705 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4229ef1e-4d03-4100-88d9-f42cd214561e-kube-api-access-q6dc8\" (UniqueName: \"kubernetes.io/projected/4229ef1e-4d03-4100-88d9-f42cd214561e-kube-api-access-q6dc8\") pod \"4229ef1e-4d03-4100-88d9-f42cd214561e\" (UID: \"4229ef1e-4d03-4100-88d9-f42cd214561e\") " Oct 30 22:23:01.697871 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.697851 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4229ef1e-4d03-4100-88d9-f42cd214561e-maas-api-openshift-ca" pod "4229ef1e-4d03-4100-88d9-f42cd214561e" (UID: "4229ef1e-4d03-4100-88d9-f42cd214561e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:01.698251 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.698232 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4229ef1e-4d03-4100-88d9-f42cd214561e-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:01.699893 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.699868 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4229ef1e-4d03-4100-88d9-f42cd214561e-kube-api-access-q6dc8" pod "4229ef1e-4d03-4100-88d9-f42cd214561e" (UID: "4229ef1e-4d03-4100-88d9-f42cd214561e"). InnerVolumeSpecName "kube-api-access-q6dc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:01.699979 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.699891 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4229ef1e-4d03-4100-88d9-f42cd214561e-maas-api-xks-ca" pod "4229ef1e-4d03-4100-88d9-f42cd214561e" (UID: "4229ef1e-4d03-4100-88d9-f42cd214561e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:01.801342 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.801305 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4229ef1e-4d03-4100-88d9-f42cd214561e-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:01.801342 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:01.801344 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6dc8\" (UniqueName: \"kubernetes.io/projected/4229ef1e-4d03-4100-88d9-f42cd214561e-kube-api-access-q6dc8\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:02.393125 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:02.393085 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s" Oct 30 22:23:02.393125 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:02.393105 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s" event={"ID":"4229ef1e-4d03-4100-88d9-f42cd214561e","Type":"ContainerDied","Data":"fb5ff6f0a4fe8eb7317999940e106789a593eb1e70eb6c45fdfef318a11dbdaf"} Oct 30 22:23:02.393339 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:02.393148 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb5ff6f0a4fe8eb7317999940e106789a593eb1e70eb6c45fdfef318a11dbdaf" Oct 30 22:23:03.058768 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:03.058737 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d13516d-0abb-4663-a016-45a93612d102" path="/var/lib/kubelet/pods/7d13516d-0abb-4663-a016-45a93612d102/volumes" Oct 30 22:23:03.059191 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:03.059175 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbecb7f4-f382-4797-aaca-f4c362171cd9" path="/var/lib/kubelet/pods/dbecb7f4-f382-4797-aaca-f4c362171cd9/volumes" Oct 30 22:23:06.958500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:06.958467 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6"] Oct 30 22:23:06.959014 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:06.958746 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" podUID="350244ea-9e66-4e83-84ff-ef81766feab1" containerName="istio-proxy" containerID="cri-o://57e4e534085573f443d0b131e4c68cba76297da253cc80863e78810f4bb72f19" gracePeriod=30 Oct 30 22:23:08.879395 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:08.879363 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx"] Oct 30 22:23:08.879878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:08.879547 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" podUID="f770114f-8048-4957-b7cd-704ffe8fafb1" containerName="istio-proxy" containerID="cri-o://fd4534770da95185d279dc90957b4785043b631afb964f2df0b3f1237745a208" gracePeriod=30 Oct 30 22:23:09.059408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.059376 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4"] Oct 30 22:23:09.059658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.059601 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" podUID="ef6c24d9-0e05-4181-9c98-dd5015f58975" containerName="istio-proxy" containerID="cri-o://e38e6d94fa6972341299b9a7913f8f299846ebc5d735f19feb304ce438a7db38" gracePeriod=30 Oct 30 22:23:09.259670 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.259590 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227"] Oct 30 22:23:09.260384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.260367 4335 cpu_manager.go:414] "containerMap: removing container" podUID="7d13516d-0abb-4663-a016-45a93612d102" containerName="payload-processing" Oct 30 22:23:09.260436 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.260386 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d13516d-0abb-4663-a016-45a93612d102" containerName="payload-processing" Oct 30 22:23:09.260436 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.260404 4335 cpu_manager.go:414] "containerMap: removing container" podUID="dbecb7f4-f382-4797-aaca-f4c362171cd9" containerName="payload-pre-processing" Oct 30 22:23:09.260436 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.260410 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbecb7f4-f382-4797-aaca-f4c362171cd9" containerName="payload-pre-processing" Oct 30 22:23:09.260436 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.260434 4335 cpu_manager.go:414] "containerMap: removing container" podUID="4229ef1e-4d03-4100-88d9-f42cd214561e" containerName="revoke-keys" Oct 30 22:23:09.260561 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.260439 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4229ef1e-4d03-4100-88d9-f42cd214561e" containerName="revoke-keys" Oct 30 22:23:09.260561 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.260546 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbecb7f4-f382-4797-aaca-f4c362171cd9" containerName="payload-pre-processing" Oct 30 22:23:09.260561 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.260560 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="4229ef1e-4d03-4100-88d9-f42cd214561e" containerName="revoke-keys" Oct 30 22:23:09.260720 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.260571 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d13516d-0abb-4663-a016-45a93612d102" containerName="payload-processing" Oct 30 22:23:09.265426 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.265408 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227" Oct 30 22:23:09.317826 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.317792 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:23:09.318041 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.318025 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:23:09.323224 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.323188 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227"] Oct 30 22:23:09.386089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.386059 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:23:09.386208 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.386131 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:23:09.456927 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.455736 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585b415c-1e7a-4d41-886c-7c33110bfc31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227\" (UID: \"585b415c-1e7a-4d41-886c-7c33110bfc31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227" Oct 30 22:23:09.456927 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.455910 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585b415c-1e7a-4d41-886c-7c33110bfc31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227\" (UID: \"585b415c-1e7a-4d41-886c-7c33110bfc31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227" Oct 30 22:23:09.456927 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.456063 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrr8d\" (UniqueName: \"kubernetes.io/projected/585b415c-1e7a-4d41-886c-7c33110bfc31-kube-api-access-vrr8d\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227\" (UID: \"585b415c-1e7a-4d41-886c-7c33110bfc31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227" Oct 30 22:23:09.557961 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.557925 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrr8d\" (UniqueName: \"kubernetes.io/projected/585b415c-1e7a-4d41-886c-7c33110bfc31-kube-api-access-vrr8d\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227\" (UID: \"585b415c-1e7a-4d41-886c-7c33110bfc31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227" Oct 30 22:23:09.558158 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.558058 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585b415c-1e7a-4d41-886c-7c33110bfc31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227\" (UID: \"585b415c-1e7a-4d41-886c-7c33110bfc31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227" Oct 30 22:23:09.558230 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.558194 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585b415c-1e7a-4d41-886c-7c33110bfc31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227\" (UID: \"585b415c-1e7a-4d41-886c-7c33110bfc31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227" Oct 30 22:23:09.558764 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.558741 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585b415c-1e7a-4d41-886c-7c33110bfc31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227\" (UID: \"585b415c-1e7a-4d41-886c-7c33110bfc31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227" Oct 30 22:23:09.560977 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.560956 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585b415c-1e7a-4d41-886c-7c33110bfc31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227\" (UID: \"585b415c-1e7a-4d41-886c-7c33110bfc31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227" Oct 30 22:23:09.619352 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.619320 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrr8d\" (UniqueName: \"kubernetes.io/projected/585b415c-1e7a-4d41-886c-7c33110bfc31-kube-api-access-vrr8d\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227\" (UID: \"585b415c-1e7a-4d41-886c-7c33110bfc31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227" Oct 30 22:23:09.879752 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:09.879678 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227" Oct 30 22:23:10.265110 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:23:10.265078 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585b415c_1e7a_4d41_886c_7c33110bfc31.slice/crio-15a810beafc02564b7812635877e3852b862a7aa5c7c1f36d92f04d3fd083d20 WatchSource:0}: Error finding container 15a810beafc02564b7812635877e3852b862a7aa5c7c1f36d92f04d3fd083d20: Status 404 returned error can't find the container with id 15a810beafc02564b7812635877e3852b862a7aa5c7c1f36d92f04d3fd083d20 Oct 30 22:23:10.265218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:10.265169 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227"] Oct 30 22:23:10.542484 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:10.542344 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227" event={"ID":"585b415c-1e7a-4d41-886c-7c33110bfc31","Type":"ContainerStarted","Data":"15a810beafc02564b7812635877e3852b862a7aa5c7c1f36d92f04d3fd083d20"} Oct 30 22:23:11.555171 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:11.555141 4335 generic.go:356] "Generic (PLEG): container finished" podID="585b415c-1e7a-4d41-886c-7c33110bfc31" containerID="e08c332d62cf3aa59bd9ec5d0b82caaea0abf022fc9aa8b23885b0ba3d9fc321" exitCode=0 Oct 30 22:23:11.555559 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:11.555223 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227" event={"ID":"585b415c-1e7a-4d41-886c-7c33110bfc31","Type":"ContainerDied","Data":"e08c332d62cf3aa59bd9ec5d0b82caaea0abf022fc9aa8b23885b0ba3d9fc321"} Oct 30 22:23:12.292417 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.292387 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:23:12.410464 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.410437 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-istio-envoy\") pod \"350244ea-9e66-4e83-84ff-ef81766feab1\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " Oct 30 22:23:12.410653 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.410498 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/350244ea-9e66-4e83-84ff-ef81766feab1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/350244ea-9e66-4e83-84ff-ef81766feab1-istio-podinfo\") pod \"350244ea-9e66-4e83-84ff-ef81766feab1\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " Oct 30 22:23:12.410653 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.410562 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-workload-certs\") pod \"350244ea-9e66-4e83-84ff-ef81766feab1\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " Oct 30 22:23:12.410783 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.410661 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-workload-socket\") pod \"350244ea-9e66-4e83-84ff-ef81766feab1\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " Oct 30 22:23:12.410783 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.410720 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/350244ea-9e66-4e83-84ff-ef81766feab1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/350244ea-9e66-4e83-84ff-ef81766feab1-istiod-ca-cert\") pod \"350244ea-9e66-4e83-84ff-ef81766feab1\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " Oct 30 22:23:12.410783 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.410772 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-istio-data\") pod \"350244ea-9e66-4e83-84ff-ef81766feab1\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " Oct 30 22:23:12.410937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.410828 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/350244ea-9e66-4e83-84ff-ef81766feab1-kube-api-access-zpkzr\" (UniqueName: \"kubernetes.io/projected/350244ea-9e66-4e83-84ff-ef81766feab1-kube-api-access-zpkzr\") pod \"350244ea-9e66-4e83-84ff-ef81766feab1\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " Oct 30 22:23:12.410937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.410881 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/350244ea-9e66-4e83-84ff-ef81766feab1-istio-token\" (UniqueName: \"kubernetes.io/projected/350244ea-9e66-4e83-84ff-ef81766feab1-istio-token\") pod \"350244ea-9e66-4e83-84ff-ef81766feab1\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " Oct 30 22:23:12.411105 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.411068 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-credential-socket\") pod \"350244ea-9e66-4e83-84ff-ef81766feab1\" (UID: \"350244ea-9e66-4e83-84ff-ef81766feab1\") " Oct 30 22:23:12.411190 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.411102 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-workload-certs" pod "350244ea-9e66-4e83-84ff-ef81766feab1" (UID: "350244ea-9e66-4e83-84ff-ef81766feab1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:12.411864 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.411760 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-workload-socket" pod "350244ea-9e66-4e83-84ff-ef81766feab1" (UID: "350244ea-9e66-4e83-84ff-ef81766feab1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:12.411864 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.411775 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-istio-data" pod "350244ea-9e66-4e83-84ff-ef81766feab1" (UID: "350244ea-9e66-4e83-84ff-ef81766feab1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:12.412060 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.411950 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-credential-socket" pod "350244ea-9e66-4e83-84ff-ef81766feab1" (UID: "350244ea-9e66-4e83-84ff-ef81766feab1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:12.412060 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.412030 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:12.412060 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.412053 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:12.412234 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.412068 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:12.412480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.412450 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/350244ea-9e66-4e83-84ff-ef81766feab1-istiod-ca-cert" pod "350244ea-9e66-4e83-84ff-ef81766feab1" (UID: "350244ea-9e66-4e83-84ff-ef81766feab1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:12.413253 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.413225 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-istio-envoy" pod "350244ea-9e66-4e83-84ff-ef81766feab1" (UID: "350244ea-9e66-4e83-84ff-ef81766feab1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:12.414306 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.414281 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/350244ea-9e66-4e83-84ff-ef81766feab1-kube-api-access-zpkzr" pod "350244ea-9e66-4e83-84ff-ef81766feab1" (UID: "350244ea-9e66-4e83-84ff-ef81766feab1"). InnerVolumeSpecName "kube-api-access-zpkzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:12.414946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.414913 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/350244ea-9e66-4e83-84ff-ef81766feab1-istio-podinfo" pod "350244ea-9e66-4e83-84ff-ef81766feab1" (UID: "350244ea-9e66-4e83-84ff-ef81766feab1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:12.415038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.415018 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/350244ea-9e66-4e83-84ff-ef81766feab1-istio-token" pod "350244ea-9e66-4e83-84ff-ef81766feab1" (UID: "350244ea-9e66-4e83-84ff-ef81766feab1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:12.513380 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.513305 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/350244ea-9e66-4e83-84ff-ef81766feab1-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:12.513380 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.513335 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpkzr\" (UniqueName: \"kubernetes.io/projected/350244ea-9e66-4e83-84ff-ef81766feab1-kube-api-access-zpkzr\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:12.513380 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.513346 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/350244ea-9e66-4e83-84ff-ef81766feab1-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:12.513380 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.513355 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:12.513380 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.513365 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/350244ea-9e66-4e83-84ff-ef81766feab1-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:12.513380 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.513373 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/350244ea-9e66-4e83-84ff-ef81766feab1-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:12.565320 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.565292 4335 generic.go:356] "Generic (PLEG): container finished" podID="350244ea-9e66-4e83-84ff-ef81766feab1" containerID="57e4e534085573f443d0b131e4c68cba76297da253cc80863e78810f4bb72f19" exitCode=0 Oct 30 22:23:12.565759 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.565344 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" Oct 30 22:23:12.565759 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.565374 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" event={"ID":"350244ea-9e66-4e83-84ff-ef81766feab1","Type":"ContainerDied","Data":"57e4e534085573f443d0b131e4c68cba76297da253cc80863e78810f4bb72f19"} Oct 30 22:23:12.565759 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.565427 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6" event={"ID":"350244ea-9e66-4e83-84ff-ef81766feab1","Type":"ContainerDied","Data":"e606f2d6d0e57bc6093ba225dfaf76621033b64580f72afe8c51aafa1e72dfbd"} Oct 30 22:23:12.565759 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.565450 4335 scope.go:122] "RemoveContainer" containerID="57e4e534085573f443d0b131e4c68cba76297da253cc80863e78810f4bb72f19" Oct 30 22:23:12.583414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.583394 4335 scope.go:122] "RemoveContainer" containerID="1dd72879d2c30c356158d8252ae367488e3cd84f0e38cbadf5946c37f3a434e5" Oct 30 22:23:12.594142 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.594125 4335 scope.go:122] "RemoveContainer" containerID="57e4e534085573f443d0b131e4c68cba76297da253cc80863e78810f4bb72f19" Oct 30 22:23:12.594514 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:23:12.594491 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e4e534085573f443d0b131e4c68cba76297da253cc80863e78810f4bb72f19\": container with ID starting with 57e4e534085573f443d0b131e4c68cba76297da253cc80863e78810f4bb72f19 not found: ID does not exist" containerID="57e4e534085573f443d0b131e4c68cba76297da253cc80863e78810f4bb72f19" Oct 30 22:23:12.594583 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.594529 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e4e534085573f443d0b131e4c68cba76297da253cc80863e78810f4bb72f19"} err="failed to get container status \"57e4e534085573f443d0b131e4c68cba76297da253cc80863e78810f4bb72f19\": rpc error: code = NotFound desc = could not find container \"57e4e534085573f443d0b131e4c68cba76297da253cc80863e78810f4bb72f19\": container with ID starting with 57e4e534085573f443d0b131e4c68cba76297da253cc80863e78810f4bb72f19 not found: ID does not exist" Oct 30 22:23:12.594583 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.594548 4335 scope.go:122] "RemoveContainer" containerID="1dd72879d2c30c356158d8252ae367488e3cd84f0e38cbadf5946c37f3a434e5" Oct 30 22:23:12.594867 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:23:12.594846 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd72879d2c30c356158d8252ae367488e3cd84f0e38cbadf5946c37f3a434e5\": container with ID starting with 1dd72879d2c30c356158d8252ae367488e3cd84f0e38cbadf5946c37f3a434e5 not found: ID does not exist" containerID="1dd72879d2c30c356158d8252ae367488e3cd84f0e38cbadf5946c37f3a434e5" Oct 30 22:23:12.594929 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.594872 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd72879d2c30c356158d8252ae367488e3cd84f0e38cbadf5946c37f3a434e5"} err="failed to get container status \"1dd72879d2c30c356158d8252ae367488e3cd84f0e38cbadf5946c37f3a434e5\": rpc error: code = NotFound desc = could not find container \"1dd72879d2c30c356158d8252ae367488e3cd84f0e38cbadf5946c37f3a434e5\": container with ID starting with 1dd72879d2c30c356158d8252ae367488e3cd84f0e38cbadf5946c37f3a434e5 not found: ID does not exist" Oct 30 22:23:12.735776 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.735727 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6"] Oct 30 22:23:12.740742 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.740712 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2d3647-openshift-default-86d4fb4cc9-zs9p6"] Oct 30 22:23:12.741056 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.741036 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227" Oct 30 22:23:12.818143 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.818120 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/585b415c-1e7a-4d41-886c-7c33110bfc31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585b415c-1e7a-4d41-886c-7c33110bfc31-maas-api-xks-ca\") pod \"585b415c-1e7a-4d41-886c-7c33110bfc31\" (UID: \"585b415c-1e7a-4d41-886c-7c33110bfc31\") " Oct 30 22:23:12.818265 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.818190 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/585b415c-1e7a-4d41-886c-7c33110bfc31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585b415c-1e7a-4d41-886c-7c33110bfc31-maas-api-openshift-ca\") pod \"585b415c-1e7a-4d41-886c-7c33110bfc31\" (UID: \"585b415c-1e7a-4d41-886c-7c33110bfc31\") " Oct 30 22:23:12.818343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.818291 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/585b415c-1e7a-4d41-886c-7c33110bfc31-kube-api-access-vrr8d\" (UniqueName: \"kubernetes.io/projected/585b415c-1e7a-4d41-886c-7c33110bfc31-kube-api-access-vrr8d\") pod \"585b415c-1e7a-4d41-886c-7c33110bfc31\" (UID: \"585b415c-1e7a-4d41-886c-7c33110bfc31\") " Oct 30 22:23:12.818579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.818526 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/585b415c-1e7a-4d41-886c-7c33110bfc31-maas-api-openshift-ca" pod "585b415c-1e7a-4d41-886c-7c33110bfc31" (UID: "585b415c-1e7a-4d41-886c-7c33110bfc31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:12.819088 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.818987 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585b415c-1e7a-4d41-886c-7c33110bfc31-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:12.820812 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.820789 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/585b415c-1e7a-4d41-886c-7c33110bfc31-maas-api-xks-ca" pod "585b415c-1e7a-4d41-886c-7c33110bfc31" (UID: "585b415c-1e7a-4d41-886c-7c33110bfc31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:12.821088 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.821070 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585b415c-1e7a-4d41-886c-7c33110bfc31-kube-api-access-vrr8d" pod "585b415c-1e7a-4d41-886c-7c33110bfc31" (UID: "585b415c-1e7a-4d41-886c-7c33110bfc31"). InnerVolumeSpecName "kube-api-access-vrr8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:12.932990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.932944 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585b415c-1e7a-4d41-886c-7c33110bfc31-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:12.932990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:12.932990 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrr8d\" (UniqueName: \"kubernetes.io/projected/585b415c-1e7a-4d41-886c-7c33110bfc31-kube-api-access-vrr8d\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:13.059669 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.059601 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="350244ea-9e66-4e83-84ff-ef81766feab1" path="/var/lib/kubelet/pods/350244ea-9e66-4e83-84ff-ef81766feab1/volumes" Oct 30 22:23:13.577006 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.576982 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227" Oct 30 22:23:13.577489 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.577021 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227" event={"ID":"585b415c-1e7a-4d41-886c-7c33110bfc31","Type":"ContainerDied","Data":"15a810beafc02564b7812635877e3852b862a7aa5c7c1f36d92f04d3fd083d20"} Oct 30 22:23:13.577489 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.577067 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15a810beafc02564b7812635877e3852b862a7aa5c7c1f36d92f04d3fd083d20" Oct 30 22:23:13.641777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.641746 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl"] Oct 30 22:23:13.642836 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.642815 4335 cpu_manager.go:414] "containerMap: removing container" podUID="585b415c-1e7a-4d41-886c-7c33110bfc31" containerName="revoke-keys" Oct 30 22:23:13.642945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.642837 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="585b415c-1e7a-4d41-886c-7c33110bfc31" containerName="revoke-keys" Oct 30 22:23:13.642945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.642880 4335 cpu_manager.go:414] "containerMap: removing container" podUID="350244ea-9e66-4e83-84ff-ef81766feab1" containerName="istio-proxy" Oct 30 22:23:13.642945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.642887 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="350244ea-9e66-4e83-84ff-ef81766feab1" containerName="istio-proxy" Oct 30 22:23:13.642945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.642904 4335 cpu_manager.go:414] "containerMap: removing container" podUID="350244ea-9e66-4e83-84ff-ef81766feab1" containerName="istio-proxy" Oct 30 22:23:13.642945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.642912 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="350244ea-9e66-4e83-84ff-ef81766feab1" containerName="istio-proxy" Oct 30 22:23:13.643179 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.643091 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="350244ea-9e66-4e83-84ff-ef81766feab1" containerName="istio-proxy" Oct 30 22:23:13.643179 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.643103 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="350244ea-9e66-4e83-84ff-ef81766feab1" containerName="istio-proxy" Oct 30 22:23:13.643179 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.643122 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="585b415c-1e7a-4d41-886c-7c33110bfc31" containerName="revoke-keys" Oct 30 22:23:13.647688 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.647666 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl" Oct 30 22:23:13.698485 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.698452 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl"] Oct 30 22:23:13.704740 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.704718 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:23:13.705018 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.704996 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:23:13.705396 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.705371 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:23:13.705497 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.705454 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:23:13.745690 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.745652 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl\" (UID: \"4489c36f-0a7c-4ade-bce8-0ff7b38cec4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl" Oct 30 22:23:13.745803 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.745784 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn8h7\" (UniqueName: \"kubernetes.io/projected/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-kube-api-access-rn8h7\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl\" (UID: \"4489c36f-0a7c-4ade-bce8-0ff7b38cec4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl" Oct 30 22:23:13.745885 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.745839 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl\" (UID: \"4489c36f-0a7c-4ade-bce8-0ff7b38cec4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl" Oct 30 22:23:13.847636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.847552 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl\" (UID: \"4489c36f-0a7c-4ade-bce8-0ff7b38cec4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl" Oct 30 22:23:13.847636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.847610 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn8h7\" (UniqueName: \"kubernetes.io/projected/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-kube-api-access-rn8h7\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl\" (UID: \"4489c36f-0a7c-4ade-bce8-0ff7b38cec4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl" Oct 30 22:23:13.847835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.847659 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl\" (UID: \"4489c36f-0a7c-4ade-bce8-0ff7b38cec4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl" Oct 30 22:23:13.848276 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.848256 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl\" (UID: \"4489c36f-0a7c-4ade-bce8-0ff7b38cec4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl" Oct 30 22:23:13.850107 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.850087 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl\" (UID: \"4489c36f-0a7c-4ade-bce8-0ff7b38cec4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl" Oct 30 22:23:13.908444 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.908420 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn8h7\" (UniqueName: \"kubernetes.io/projected/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-kube-api-access-rn8h7\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl\" (UID: \"4489c36f-0a7c-4ade-bce8-0ff7b38cec4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl" Oct 30 22:23:13.963131 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:13.963105 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl" Oct 30 22:23:14.240284 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.240258 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:23:14.256216 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.256150 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-workload-socket\") pod \"f770114f-8048-4957-b7cd-704ffe8fafb1\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " Oct 30 22:23:14.256216 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.256207 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-workload-certs\") pod \"f770114f-8048-4957-b7cd-704ffe8fafb1\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " Oct 30 22:23:14.257430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.256244 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-envoy\") pod \"f770114f-8048-4957-b7cd-704ffe8fafb1\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " Oct 30 22:23:14.257430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.256296 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f770114f-8048-4957-b7cd-704ffe8fafb1-kube-api-access-n58vv\" (UniqueName: \"kubernetes.io/projected/f770114f-8048-4957-b7cd-704ffe8fafb1-kube-api-access-n58vv\") pod \"f770114f-8048-4957-b7cd-704ffe8fafb1\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " Oct 30 22:23:14.257430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.256338 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-data\") pod \"f770114f-8048-4957-b7cd-704ffe8fafb1\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " Oct 30 22:23:14.257430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.256367 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-token\" (UniqueName: \"kubernetes.io/projected/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-token\") pod \"f770114f-8048-4957-b7cd-704ffe8fafb1\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " Oct 30 22:23:14.257430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.256393 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-credential-socket\") pod \"f770114f-8048-4957-b7cd-704ffe8fafb1\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " Oct 30 22:23:14.257430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.256489 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-podinfo\") pod \"f770114f-8048-4957-b7cd-704ffe8fafb1\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " Oct 30 22:23:14.257430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.256531 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f770114f-8048-4957-b7cd-704ffe8fafb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f770114f-8048-4957-b7cd-704ffe8fafb1-istiod-ca-cert\") pod \"f770114f-8048-4957-b7cd-704ffe8fafb1\" (UID: \"f770114f-8048-4957-b7cd-704ffe8fafb1\") " Oct 30 22:23:14.257430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.257196 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-data" pod "f770114f-8048-4957-b7cd-704ffe8fafb1" (UID: "f770114f-8048-4957-b7cd-704ffe8fafb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:14.257430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.257418 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f770114f-8048-4957-b7cd-704ffe8fafb1-istiod-ca-cert" pod "f770114f-8048-4957-b7cd-704ffe8fafb1" (UID: "f770114f-8048-4957-b7cd-704ffe8fafb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:14.257966 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.257775 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-credential-socket" pod "f770114f-8048-4957-b7cd-704ffe8fafb1" (UID: "f770114f-8048-4957-b7cd-704ffe8fafb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:14.257966 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.257801 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-workload-socket" pod "f770114f-8048-4957-b7cd-704ffe8fafb1" (UID: "f770114f-8048-4957-b7cd-704ffe8fafb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:14.257966 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.257881 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-workload-certs" pod "f770114f-8048-4957-b7cd-704ffe8fafb1" (UID: "f770114f-8048-4957-b7cd-704ffe8fafb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:14.260799 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.260658 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f770114f-8048-4957-b7cd-704ffe8fafb1-kube-api-access-n58vv" pod "f770114f-8048-4957-b7cd-704ffe8fafb1" (UID: "f770114f-8048-4957-b7cd-704ffe8fafb1"). InnerVolumeSpecName "kube-api-access-n58vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:14.261080 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.261062 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-envoy" pod "f770114f-8048-4957-b7cd-704ffe8fafb1" (UID: "f770114f-8048-4957-b7cd-704ffe8fafb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:14.263147 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.263126 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-podinfo" pod "f770114f-8048-4957-b7cd-704ffe8fafb1" (UID: "f770114f-8048-4957-b7cd-704ffe8fafb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:14.263730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.263711 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-token" pod "f770114f-8048-4957-b7cd-704ffe8fafb1" (UID: "f770114f-8048-4957-b7cd-704ffe8fafb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:14.339856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.339828 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:23:14.350967 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:23:14.350934 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4489c36f_0a7c_4ade_bce8_0ff7b38cec4b.slice/crio-ed75575d7862a7696da425e17e3ce3420fb2eedf5a3bb3f99318fee16455a67c WatchSource:0}: Error finding container ed75575d7862a7696da425e17e3ce3420fb2eedf5a3bb3f99318fee16455a67c: Status 404 returned error can't find the container with id ed75575d7862a7696da425e17e3ce3420fb2eedf5a3bb3f99318fee16455a67c Oct 30 22:23:14.351199 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.351180 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl"] Oct 30 22:23:14.357734 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.357700 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-workload-certs\") pod \"ef6c24d9-0e05-4181-9c98-dd5015f58975\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " Oct 30 22:23:14.357838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.357746 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-credential-socket\") pod \"ef6c24d9-0e05-4181-9c98-dd5015f58975\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " Oct 30 22:23:14.357838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.357787 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-token\" (UniqueName: \"kubernetes.io/projected/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-token\") pod \"ef6c24d9-0e05-4181-9c98-dd5015f58975\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " Oct 30 22:23:14.357944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.357859 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef6c24d9-0e05-4181-9c98-dd5015f58975-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6c24d9-0e05-4181-9c98-dd5015f58975-istiod-ca-cert\") pod \"ef6c24d9-0e05-4181-9c98-dd5015f58975\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " Oct 30 22:23:14.357944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.357941 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-envoy\") pod \"ef6c24d9-0e05-4181-9c98-dd5015f58975\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " Oct 30 22:23:14.358044 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.357962 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-data\") pod \"ef6c24d9-0e05-4181-9c98-dd5015f58975\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " Oct 30 22:23:14.358044 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.357981 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-workload-certs" pod "ef6c24d9-0e05-4181-9c98-dd5015f58975" (UID: "ef6c24d9-0e05-4181-9c98-dd5015f58975"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:14.358044 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.358003 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-workload-socket\") pod \"ef6c24d9-0e05-4181-9c98-dd5015f58975\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " Oct 30 22:23:14.358181 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.358078 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6c24d9-0e05-4181-9c98-dd5015f58975-kube-api-access-bfqts\" (UniqueName: \"kubernetes.io/projected/ef6c24d9-0e05-4181-9c98-dd5015f58975-kube-api-access-bfqts\") pod \"ef6c24d9-0e05-4181-9c98-dd5015f58975\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " Oct 30 22:23:14.358181 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.358144 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-podinfo\") pod \"ef6c24d9-0e05-4181-9c98-dd5015f58975\" (UID: \"ef6c24d9-0e05-4181-9c98-dd5015f58975\") " Oct 30 22:23:14.358386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.358350 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-credential-socket" pod "ef6c24d9-0e05-4181-9c98-dd5015f58975" (UID: "ef6c24d9-0e05-4181-9c98-dd5015f58975"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:14.358386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.358277 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6c24d9-0e05-4181-9c98-dd5015f58975-istiod-ca-cert" pod "ef6c24d9-0e05-4181-9c98-dd5015f58975" (UID: "ef6c24d9-0e05-4181-9c98-dd5015f58975"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:14.358386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.358366 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-data" pod "ef6c24d9-0e05-4181-9c98-dd5015f58975" (UID: "ef6c24d9-0e05-4181-9c98-dd5015f58975"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:14.358637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.358423 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-workload-socket" pod "ef6c24d9-0e05-4181-9c98-dd5015f58975" (UID: "ef6c24d9-0e05-4181-9c98-dd5015f58975"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:14.358835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.358811 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:14.359100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.358841 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:14.359100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.358857 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:14.359100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.358872 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:14.359100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.358885 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:14.359100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.358899 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:14.359100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.358913 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f770114f-8048-4957-b7cd-704ffe8fafb1-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:14.359100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.358965 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:14.359100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.359069 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:14.359568 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.359115 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:14.359568 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.359132 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:14.359568 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.359146 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f770114f-8048-4957-b7cd-704ffe8fafb1-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:14.359568 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.359160 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6c24d9-0e05-4181-9c98-dd5015f58975-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:14.359568 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.359175 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n58vv\" (UniqueName: \"kubernetes.io/projected/f770114f-8048-4957-b7cd-704ffe8fafb1-kube-api-access-n58vv\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:14.360557 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.360534 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-podinfo" pod "ef6c24d9-0e05-4181-9c98-dd5015f58975" (UID: "ef6c24d9-0e05-4181-9c98-dd5015f58975"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:14.360699 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.360558 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-token" pod "ef6c24d9-0e05-4181-9c98-dd5015f58975" (UID: "ef6c24d9-0e05-4181-9c98-dd5015f58975"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:14.360770 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.360753 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6c24d9-0e05-4181-9c98-dd5015f58975-kube-api-access-bfqts" pod "ef6c24d9-0e05-4181-9c98-dd5015f58975" (UID: "ef6c24d9-0e05-4181-9c98-dd5015f58975"). InnerVolumeSpecName "kube-api-access-bfqts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:14.362244 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.362219 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-envoy" pod "ef6c24d9-0e05-4181-9c98-dd5015f58975" (UID: "ef6c24d9-0e05-4181-9c98-dd5015f58975"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:14.460367 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.460332 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:14.460367 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.460360 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfqts\" (UniqueName: \"kubernetes.io/projected/ef6c24d9-0e05-4181-9c98-dd5015f58975-kube-api-access-bfqts\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:14.460367 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.460372 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:14.460584 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.460381 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6c24d9-0e05-4181-9c98-dd5015f58975-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:14.586305 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.586260 4335 generic.go:356] "Generic (PLEG): container finished" podID="ef6c24d9-0e05-4181-9c98-dd5015f58975" containerID="e38e6d94fa6972341299b9a7913f8f299846ebc5d735f19feb304ce438a7db38" exitCode=0 Oct 30 22:23:14.586769 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.586315 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" Oct 30 22:23:14.586769 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.586346 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" event={"ID":"ef6c24d9-0e05-4181-9c98-dd5015f58975","Type":"ContainerDied","Data":"e38e6d94fa6972341299b9a7913f8f299846ebc5d735f19feb304ce438a7db38"} Oct 30 22:23:14.586769 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.586400 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4" event={"ID":"ef6c24d9-0e05-4181-9c98-dd5015f58975","Type":"ContainerDied","Data":"e28ce7add067438abdfab973f3711c36b0c080280e15b085177c179d7863c9a4"} Oct 30 22:23:14.586769 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.586424 4335 scope.go:122] "RemoveContainer" containerID="e38e6d94fa6972341299b9a7913f8f299846ebc5d735f19feb304ce438a7db38" Oct 30 22:23:14.588276 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.588251 4335 generic.go:356] "Generic (PLEG): container finished" podID="f770114f-8048-4957-b7cd-704ffe8fafb1" containerID="fd4534770da95185d279dc90957b4785043b631afb964f2df0b3f1237745a208" exitCode=0 Oct 30 22:23:14.588393 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.588292 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" Oct 30 22:23:14.588393 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.588340 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" event={"ID":"f770114f-8048-4957-b7cd-704ffe8fafb1","Type":"ContainerDied","Data":"fd4534770da95185d279dc90957b4785043b631afb964f2df0b3f1237745a208"} Oct 30 22:23:14.588393 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.588383 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx" event={"ID":"f770114f-8048-4957-b7cd-704ffe8fafb1","Type":"ContainerDied","Data":"e0880ce5f2e241c02c4e30eee55baf4268f3a39f50b1dd8996f01144a2bf408e"} Oct 30 22:23:14.598086 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.597729 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl" event={"ID":"4489c36f-0a7c-4ade-bce8-0ff7b38cec4b","Type":"ContainerStarted","Data":"ed75575d7862a7696da425e17e3ce3420fb2eedf5a3bb3f99318fee16455a67c"} Oct 30 22:23:14.608552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.608535 4335 scope.go:122] "RemoveContainer" containerID="e38e6d94fa6972341299b9a7913f8f299846ebc5d735f19feb304ce438a7db38" Oct 30 22:23:14.608880 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:23:14.608857 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38e6d94fa6972341299b9a7913f8f299846ebc5d735f19feb304ce438a7db38\": container with ID starting with e38e6d94fa6972341299b9a7913f8f299846ebc5d735f19feb304ce438a7db38 not found: ID does not exist" containerID="e38e6d94fa6972341299b9a7913f8f299846ebc5d735f19feb304ce438a7db38" Oct 30 22:23:14.608958 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.608892 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38e6d94fa6972341299b9a7913f8f299846ebc5d735f19feb304ce438a7db38"} err="failed to get container status \"e38e6d94fa6972341299b9a7913f8f299846ebc5d735f19feb304ce438a7db38\": rpc error: code = NotFound desc = could not find container \"e38e6d94fa6972341299b9a7913f8f299846ebc5d735f19feb304ce438a7db38\": container with ID starting with e38e6d94fa6972341299b9a7913f8f299846ebc5d735f19feb304ce438a7db38 not found: ID does not exist" Oct 30 22:23:14.608958 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.608911 4335 scope.go:122] "RemoveContainer" containerID="fd4534770da95185d279dc90957b4785043b631afb964f2df0b3f1237745a208" Oct 30 22:23:14.619344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.619324 4335 scope.go:122] "RemoveContainer" containerID="fd4534770da95185d279dc90957b4785043b631afb964f2df0b3f1237745a208" Oct 30 22:23:14.619650 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:23:14.619609 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd4534770da95185d279dc90957b4785043b631afb964f2df0b3f1237745a208\": container with ID starting with fd4534770da95185d279dc90957b4785043b631afb964f2df0b3f1237745a208 not found: ID does not exist" containerID="fd4534770da95185d279dc90957b4785043b631afb964f2df0b3f1237745a208" Oct 30 22:23:14.619729 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.619660 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd4534770da95185d279dc90957b4785043b631afb964f2df0b3f1237745a208"} err="failed to get container status \"fd4534770da95185d279dc90957b4785043b631afb964f2df0b3f1237745a208\": rpc error: code = NotFound desc = could not find container \"fd4534770da95185d279dc90957b4785043b631afb964f2df0b3f1237745a208\": container with ID starting with fd4534770da95185d279dc90957b4785043b631afb964f2df0b3f1237745a208 not found: ID does not exist" Oct 30 22:23:14.765667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.765625 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4"] Oct 30 22:23:14.771222 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.771200 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-28sf4"] Oct 30 22:23:14.869967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.869887 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx"] Oct 30 22:23:14.874375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:14.874353 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-rrvfx"] Oct 30 22:23:15.059312 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:15.059283 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6c24d9-0e05-4181-9c98-dd5015f58975" path="/var/lib/kubelet/pods/ef6c24d9-0e05-4181-9c98-dd5015f58975/volumes" Oct 30 22:23:15.059869 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:15.059792 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f770114f-8048-4957-b7cd-704ffe8fafb1" path="/var/lib/kubelet/pods/f770114f-8048-4957-b7cd-704ffe8fafb1/volumes" Oct 30 22:23:15.614759 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:15.614721 4335 generic.go:356] "Generic (PLEG): container finished" podID="4489c36f-0a7c-4ade-bce8-0ff7b38cec4b" containerID="ea858b00cd099926dc7c61b6c79579c04944d7f4e13527f55c4b726ed60c77dd" exitCode=0 Oct 30 22:23:15.615240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:15.614806 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl" event={"ID":"4489c36f-0a7c-4ade-bce8-0ff7b38cec4b","Type":"ContainerDied","Data":"ea858b00cd099926dc7c61b6c79579c04944d7f4e13527f55c4b726ed60c77dd"} Oct 30 22:23:16.050627 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:16.050594 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj"] Oct 30 22:23:16.055003 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:16.054973 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2298aa0-vffcj"] Oct 30 22:23:16.789186 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:16.789161 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl" Oct 30 22:23:16.888201 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:16.888138 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-maas-api-xks-ca\") pod \"4489c36f-0a7c-4ade-bce8-0ff7b38cec4b\" (UID: \"4489c36f-0a7c-4ade-bce8-0ff7b38cec4b\") " Oct 30 22:23:16.888351 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:16.888243 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-kube-api-access-rn8h7\" (UniqueName: \"kubernetes.io/projected/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-kube-api-access-rn8h7\") pod \"4489c36f-0a7c-4ade-bce8-0ff7b38cec4b\" (UID: \"4489c36f-0a7c-4ade-bce8-0ff7b38cec4b\") " Oct 30 22:23:16.888351 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:16.888270 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-maas-api-openshift-ca\") pod \"4489c36f-0a7c-4ade-bce8-0ff7b38cec4b\" (UID: \"4489c36f-0a7c-4ade-bce8-0ff7b38cec4b\") " Oct 30 22:23:16.888741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:16.888713 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-maas-api-openshift-ca" pod "4489c36f-0a7c-4ade-bce8-0ff7b38cec4b" (UID: "4489c36f-0a7c-4ade-bce8-0ff7b38cec4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:16.888963 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:16.888914 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:16.891078 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:16.891051 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-maas-api-xks-ca" pod "4489c36f-0a7c-4ade-bce8-0ff7b38cec4b" (UID: "4489c36f-0a7c-4ade-bce8-0ff7b38cec4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:16.891165 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:16.891133 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-kube-api-access-rn8h7" pod "4489c36f-0a7c-4ade-bce8-0ff7b38cec4b" (UID: "4489c36f-0a7c-4ade-bce8-0ff7b38cec4b"). InnerVolumeSpecName "kube-api-access-rn8h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:16.990002 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:16.989971 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn8h7\" (UniqueName: \"kubernetes.io/projected/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-kube-api-access-rn8h7\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:16.990002 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:16.989999 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:17.059725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:17.059693 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1122f4c8-84b1-46f0-825e-a51e2fb85b73" path="/var/lib/kubelet/pods/1122f4c8-84b1-46f0-825e-a51e2fb85b73/volumes" Oct 30 22:23:17.639218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:17.639178 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl" Oct 30 22:23:17.639396 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:17.639284 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl" event={"ID":"4489c36f-0a7c-4ade-bce8-0ff7b38cec4b","Type":"ContainerDied","Data":"ed75575d7862a7696da425e17e3ce3420fb2eedf5a3bb3f99318fee16455a67c"} Oct 30 22:23:17.639396 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:17.639339 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed75575d7862a7696da425e17e3ce3420fb2eedf5a3bb3f99318fee16455a67c" Oct 30 22:23:19.661753 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:19.661717 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s"] Oct 30 22:23:19.710707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:19.710677 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a78655-5776s"] Oct 30 22:23:20.048692 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:20.048660 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz"] Oct 30 22:23:21.058956 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:21.058920 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4229ef1e-4d03-4100-88d9-f42cd214561e" path="/var/lib/kubelet/pods/4229ef1e-4d03-4100-88d9-f42cd214561e/volumes" Oct 30 22:23:24.035384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:24.035348 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv"] Oct 30 22:23:24.153879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:24.153850 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb"] Oct 30 22:23:25.049721 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:25.049673 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz" podUID="94191978-6460-43f3-9799-42c7c6286b2a" containerName="payload-processing" containerID="cri-o://f174e2cd9d2ffbc97d9bcd81d8345ede1bc7a9ec25ede5c39d0a2abffdbcf158" gracePeriod=25 Oct 30 22:23:25.371718 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:23:25.371684 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:23:25.507283 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:25.506912 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz" Oct 30 22:23:25.683285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:25.683200 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94191978-6460-43f3-9799-42c7c6286b2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94191978-6460-43f3-9799-42c7c6286b2a-plugins-config-volume\") pod \"94191978-6460-43f3-9799-42c7c6286b2a\" (UID: \"94191978-6460-43f3-9799-42c7c6286b2a\") " Oct 30 22:23:25.683453 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:25.683299 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94191978-6460-43f3-9799-42c7c6286b2a-kube-api-access-gqhvf\" (UniqueName: \"kubernetes.io/projected/94191978-6460-43f3-9799-42c7c6286b2a-kube-api-access-gqhvf\") pod \"94191978-6460-43f3-9799-42c7c6286b2a\" (UID: \"94191978-6460-43f3-9799-42c7c6286b2a\") " Oct 30 22:23:25.683593 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:25.683570 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94191978-6460-43f3-9799-42c7c6286b2a-plugins-config-volume" pod "94191978-6460-43f3-9799-42c7c6286b2a" (UID: "94191978-6460-43f3-9799-42c7c6286b2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:25.683859 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:25.683842 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94191978-6460-43f3-9799-42c7c6286b2a-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:25.685812 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:25.685778 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94191978-6460-43f3-9799-42c7c6286b2a-kube-api-access-gqhvf" pod "94191978-6460-43f3-9799-42c7c6286b2a" (UID: "94191978-6460-43f3-9799-42c7c6286b2a"). InnerVolumeSpecName "kube-api-access-gqhvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:25.765850 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:25.765814 4335 generic.go:356] "Generic (PLEG): container finished" podID="94191978-6460-43f3-9799-42c7c6286b2a" containerID="f174e2cd9d2ffbc97d9bcd81d8345ede1bc7a9ec25ede5c39d0a2abffdbcf158" exitCode=0 Oct 30 22:23:25.766116 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:25.765873 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz" Oct 30 22:23:25.766267 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:25.765896 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz" event={"ID":"94191978-6460-43f3-9799-42c7c6286b2a","Type":"ContainerDied","Data":"f174e2cd9d2ffbc97d9bcd81d8345ede1bc7a9ec25ede5c39d0a2abffdbcf158"} Oct 30 22:23:25.766347 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:25.766289 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz" event={"ID":"94191978-6460-43f3-9799-42c7c6286b2a","Type":"ContainerDied","Data":"e1ee23b5e62f5047ed6344e18d8515cef8e24b7ff11a2f405d2dbb5834c40c0c"} Oct 30 22:23:25.766347 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:25.766325 4335 scope.go:122] "RemoveContainer" containerID="f174e2cd9d2ffbc97d9bcd81d8345ede1bc7a9ec25ede5c39d0a2abffdbcf158" Oct 30 22:23:25.780506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:25.780490 4335 scope.go:122] "RemoveContainer" containerID="f174e2cd9d2ffbc97d9bcd81d8345ede1bc7a9ec25ede5c39d0a2abffdbcf158" Oct 30 22:23:25.780858 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:23:25.780834 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f174e2cd9d2ffbc97d9bcd81d8345ede1bc7a9ec25ede5c39d0a2abffdbcf158\": container with ID starting with f174e2cd9d2ffbc97d9bcd81d8345ede1bc7a9ec25ede5c39d0a2abffdbcf158 not found: ID does not exist" containerID="f174e2cd9d2ffbc97d9bcd81d8345ede1bc7a9ec25ede5c39d0a2abffdbcf158" Oct 30 22:23:25.780942 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:25.780868 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f174e2cd9d2ffbc97d9bcd81d8345ede1bc7a9ec25ede5c39d0a2abffdbcf158"} err="failed to get container status \"f174e2cd9d2ffbc97d9bcd81d8345ede1bc7a9ec25ede5c39d0a2abffdbcf158\": rpc error: code = NotFound desc = could not find container \"f174e2cd9d2ffbc97d9bcd81d8345ede1bc7a9ec25ede5c39d0a2abffdbcf158\": container with ID starting with f174e2cd9d2ffbc97d9bcd81d8345ede1bc7a9ec25ede5c39d0a2abffdbcf158 not found: ID does not exist" Oct 30 22:23:25.785386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:25.785360 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqhvf\" (UniqueName: \"kubernetes.io/projected/94191978-6460-43f3-9799-42c7c6286b2a-kube-api-access-gqhvf\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:25.936519 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:25.936435 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz"] Oct 30 22:23:25.941411 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:25.941382 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2d8852-7b44fcbfb9-tzsgz"] Oct 30 22:23:27.139431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:27.139397 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94191978-6460-43f3-9799-42c7c6286b2a" path="/var/lib/kubelet/pods/94191978-6460-43f3-9799-42c7c6286b2a/volumes" Oct 30 22:23:29.035675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.035600 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv" podUID="5c18511f-79df-44be-807e-f8d2744c7949" containerName="payload-processing" containerID="cri-o://4886f68da238f2d21e3a9902c3f508592c4c273265378beee5b971c6bb54a97c" gracePeriod=25 Oct 30 22:23:29.109064 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:23:29.108995 4335 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c18511f_79df_44be_807e_f8d2744c7949.slice/crio-4886f68da238f2d21e3a9902c3f508592c4c273265378beee5b971c6bb54a97c.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:23:29.155193 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.155120 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb" podUID="fcfccb62-83d1-40d8-b0e4-8e04c9319490" containerName="payload-pre-processing" containerID="cri-o://1f9df4088b1d2dc85a51e60a4a26aff77b23fa4b84891fb1bb5be2ed856637ba" gracePeriod=25 Oct 30 22:23:29.349873 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.349841 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl"] Oct 30 22:23:29.359606 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.359521 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv" Oct 30 22:23:29.397165 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.397123 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ec058c-fj6dl"] Oct 30 22:23:29.468881 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.468849 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c18511f-79df-44be-807e-f8d2744c7949-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c18511f-79df-44be-807e-f8d2744c7949-plugins-config-volume\") pod \"5c18511f-79df-44be-807e-f8d2744c7949\" (UID: \"5c18511f-79df-44be-807e-f8d2744c7949\") " Oct 30 22:23:29.469106 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.469084 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c18511f-79df-44be-807e-f8d2744c7949-kube-api-access-7sk67\" (UniqueName: \"kubernetes.io/projected/5c18511f-79df-44be-807e-f8d2744c7949-kube-api-access-7sk67\") pod \"5c18511f-79df-44be-807e-f8d2744c7949\" (UID: \"5c18511f-79df-44be-807e-f8d2744c7949\") " Oct 30 22:23:29.469466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.469220 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c18511f-79df-44be-807e-f8d2744c7949-plugins-config-volume" pod "5c18511f-79df-44be-807e-f8d2744c7949" (UID: "5c18511f-79df-44be-807e-f8d2744c7949"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:29.470882 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.469806 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c18511f-79df-44be-807e-f8d2744c7949-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:29.471369 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.471340 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c18511f-79df-44be-807e-f8d2744c7949-kube-api-access-7sk67" pod "5c18511f-79df-44be-807e-f8d2744c7949" (UID: "5c18511f-79df-44be-807e-f8d2744c7949"). InnerVolumeSpecName "kube-api-access-7sk67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:29.491630 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.491586 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb" Oct 30 22:23:29.570715 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.570689 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcfccb62-83d1-40d8-b0e4-8e04c9319490-kube-api-access-nvm4w\" (UniqueName: \"kubernetes.io/projected/fcfccb62-83d1-40d8-b0e4-8e04c9319490-kube-api-access-nvm4w\") pod \"fcfccb62-83d1-40d8-b0e4-8e04c9319490\" (UID: \"fcfccb62-83d1-40d8-b0e4-8e04c9319490\") " Oct 30 22:23:29.570842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.570830 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcfccb62-83d1-40d8-b0e4-8e04c9319490-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfccb62-83d1-40d8-b0e4-8e04c9319490-plugins-config-volume\") pod \"fcfccb62-83d1-40d8-b0e4-8e04c9319490\" (UID: \"fcfccb62-83d1-40d8-b0e4-8e04c9319490\") " Oct 30 22:23:29.571180 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.571157 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfccb62-83d1-40d8-b0e4-8e04c9319490-plugins-config-volume" pod "fcfccb62-83d1-40d8-b0e4-8e04c9319490" (UID: "fcfccb62-83d1-40d8-b0e4-8e04c9319490"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:29.571343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.571299 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sk67\" (UniqueName: \"kubernetes.io/projected/5c18511f-79df-44be-807e-f8d2744c7949-kube-api-access-7sk67\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:29.571343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.571320 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfccb62-83d1-40d8-b0e4-8e04c9319490-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:29.572898 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.572880 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfccb62-83d1-40d8-b0e4-8e04c9319490-kube-api-access-nvm4w" pod "fcfccb62-83d1-40d8-b0e4-8e04c9319490" (UID: "fcfccb62-83d1-40d8-b0e4-8e04c9319490"). InnerVolumeSpecName "kube-api-access-nvm4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:29.673142 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.673074 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvm4w\" (UniqueName: \"kubernetes.io/projected/fcfccb62-83d1-40d8-b0e4-8e04c9319490-kube-api-access-nvm4w\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:29.811927 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.811888 4335 generic.go:356] "Generic (PLEG): container finished" podID="fcfccb62-83d1-40d8-b0e4-8e04c9319490" containerID="1f9df4088b1d2dc85a51e60a4a26aff77b23fa4b84891fb1bb5be2ed856637ba" exitCode=0 Oct 30 22:23:29.812104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.811950 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb" Oct 30 22:23:29.812104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.811965 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb" event={"ID":"fcfccb62-83d1-40d8-b0e4-8e04c9319490","Type":"ContainerDied","Data":"1f9df4088b1d2dc85a51e60a4a26aff77b23fa4b84891fb1bb5be2ed856637ba"} Oct 30 22:23:29.812104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.812026 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb" event={"ID":"fcfccb62-83d1-40d8-b0e4-8e04c9319490","Type":"ContainerDied","Data":"7c36fb020de579307ea83de5676859cae962c31423116ba62154a717c3b66dbf"} Oct 30 22:23:29.812104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.812057 4335 scope.go:122] "RemoveContainer" containerID="1f9df4088b1d2dc85a51e60a4a26aff77b23fa4b84891fb1bb5be2ed856637ba" Oct 30 22:23:29.817919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.814535 4335 generic.go:356] "Generic (PLEG): container finished" podID="5c18511f-79df-44be-807e-f8d2744c7949" containerID="4886f68da238f2d21e3a9902c3f508592c4c273265378beee5b971c6bb54a97c" exitCode=0 Oct 30 22:23:29.817919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.814595 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv" Oct 30 22:23:29.817919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.814773 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv" event={"ID":"5c18511f-79df-44be-807e-f8d2744c7949","Type":"ContainerDied","Data":"4886f68da238f2d21e3a9902c3f508592c4c273265378beee5b971c6bb54a97c"} Oct 30 22:23:29.817919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.814849 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv" event={"ID":"5c18511f-79df-44be-807e-f8d2744c7949","Type":"ContainerDied","Data":"7ac287e7515c22d333e717cfe4ce6011f0b2bdd8624c405ad21a1a29f35ffb86"} Oct 30 22:23:29.832203 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.832182 4335 scope.go:122] "RemoveContainer" containerID="1f9df4088b1d2dc85a51e60a4a26aff77b23fa4b84891fb1bb5be2ed856637ba" Oct 30 22:23:29.832737 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:23:29.832711 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f9df4088b1d2dc85a51e60a4a26aff77b23fa4b84891fb1bb5be2ed856637ba\": container with ID starting with 1f9df4088b1d2dc85a51e60a4a26aff77b23fa4b84891fb1bb5be2ed856637ba not found: ID does not exist" containerID="1f9df4088b1d2dc85a51e60a4a26aff77b23fa4b84891fb1bb5be2ed856637ba" Oct 30 22:23:29.833225 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.833145 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9df4088b1d2dc85a51e60a4a26aff77b23fa4b84891fb1bb5be2ed856637ba"} err="failed to get container status \"1f9df4088b1d2dc85a51e60a4a26aff77b23fa4b84891fb1bb5be2ed856637ba\": rpc error: code = NotFound desc = could not find container \"1f9df4088b1d2dc85a51e60a4a26aff77b23fa4b84891fb1bb5be2ed856637ba\": container with ID starting with 1f9df4088b1d2dc85a51e60a4a26aff77b23fa4b84891fb1bb5be2ed856637ba not found: ID does not exist" Oct 30 22:23:29.833414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.833399 4335 scope.go:122] "RemoveContainer" containerID="4886f68da238f2d21e3a9902c3f508592c4c273265378beee5b971c6bb54a97c" Oct 30 22:23:29.852830 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.852810 4335 scope.go:122] "RemoveContainer" containerID="4886f68da238f2d21e3a9902c3f508592c4c273265378beee5b971c6bb54a97c" Oct 30 22:23:29.853158 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:23:29.853136 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4886f68da238f2d21e3a9902c3f508592c4c273265378beee5b971c6bb54a97c\": container with ID starting with 4886f68da238f2d21e3a9902c3f508592c4c273265378beee5b971c6bb54a97c not found: ID does not exist" containerID="4886f68da238f2d21e3a9902c3f508592c4c273265378beee5b971c6bb54a97c" Oct 30 22:23:29.853252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.853166 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4886f68da238f2d21e3a9902c3f508592c4c273265378beee5b971c6bb54a97c"} err="failed to get container status \"4886f68da238f2d21e3a9902c3f508592c4c273265378beee5b971c6bb54a97c\": rpc error: code = NotFound desc = could not find container \"4886f68da238f2d21e3a9902c3f508592c4c273265378beee5b971c6bb54a97c\": container with ID starting with 4886f68da238f2d21e3a9902c3f508592c4c273265378beee5b971c6bb54a97c not found: ID does not exist" Oct 30 22:23:29.993316 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:29.993227 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb"] Oct 30 22:23:30.006292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:30.006260 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ec058c-6698446547-dtnrb"] Oct 30 22:23:30.126267 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:30.126234 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv"] Oct 30 22:23:30.138388 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:30.138357 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ec058c-7c8bd9cf89-gcvjv"] Oct 30 22:23:31.059745 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:31.059713 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4489c36f-0a7c-4ade-bce8-0ff7b38cec4b" path="/var/lib/kubelet/pods/4489c36f-0a7c-4ade-bce8-0ff7b38cec4b/volumes" Oct 30 22:23:31.060159 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:31.060140 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c18511f-79df-44be-807e-f8d2744c7949" path="/var/lib/kubelet/pods/5c18511f-79df-44be-807e-f8d2744c7949/volumes" Oct 30 22:23:31.060514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:31.060498 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcfccb62-83d1-40d8-b0e4-8e04c9319490" path="/var/lib/kubelet/pods/fcfccb62-83d1-40d8-b0e4-8e04c9319490/volumes" Oct 30 22:23:33.861405 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:33.861369 4335 generic.go:356] "Generic (PLEG): container finished" podID="4e974fd0-2c92-4626-9673-d202cc596fea" containerID="d9da18cde1e801e291e005dd56c19ad7a9520de55b4d855408e7a44e5f847367" exitCode=0 Oct 30 22:23:33.861922 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:33.861441 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" event={"ID":"4e974fd0-2c92-4626-9673-d202cc596fea","Type":"ContainerDied","Data":"d9da18cde1e801e291e005dd56c19ad7a9520de55b4d855408e7a44e5f847367"} Oct 30 22:23:33.861922 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:33.861793 4335 scope.go:122] "RemoveContainer" containerID="d9da18cde1e801e291e005dd56c19ad7a9520de55b4d855408e7a44e5f847367" Oct 30 22:23:33.863299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:33.863272 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:23:33.863376 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:33.863342 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:23:33.863413 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:33.863375 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:23:33.981163 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:33.981133 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:23:34.872894 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:34.872858 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" event={"ID":"4e974fd0-2c92-4626-9673-d202cc596fea","Type":"ContainerStarted","Data":"48acb51650dc8ee47ddc693c72a531b1e0656949e2ee57f0b530084ff069dd55"} Oct 30 22:23:34.933730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:34.933696 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:23:34.933944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:34.933882 4335 patch_prober.go:29] interesting pod/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 22:23:34.933944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:34.933921 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" podUID="4e974fd0-2c92-4626-9673-d202cc596fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 22:23:35.932816 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:35.932777 4335 patch_prober.go:29] interesting pod/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 22:23:35.933326 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:35.932852 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" podUID="4e974fd0-2c92-4626-9673-d202cc596fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 22:23:36.747056 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:36.746924 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p"] Oct 30 22:23:36.747563 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:36.747530 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" podUID="e7f63a86-2ce4-45ad-8f6a-38025710f1f5" containerName="istio-proxy" containerID="cri-o://5642d47bd7500a209cf89862e77d3baee4c4bbc672c30398736b944b1374d51a" gracePeriod=30 Oct 30 22:23:36.940959 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:36.940356 4335 patch_prober.go:29] interesting pod/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 22:23:36.940959 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:36.940425 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" podUID="4e974fd0-2c92-4626-9673-d202cc596fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 22:23:37.661027 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.660985 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w"] Oct 30 22:23:37.661737 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.661720 4335 cpu_manager.go:414] "containerMap: removing container" podUID="4489c36f-0a7c-4ade-bce8-0ff7b38cec4b" containerName="revoke-keys" Oct 30 22:23:37.661737 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.661737 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4489c36f-0a7c-4ade-bce8-0ff7b38cec4b" containerName="revoke-keys" Oct 30 22:23:37.661838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.661754 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f770114f-8048-4957-b7cd-704ffe8fafb1" containerName="istio-proxy" Oct 30 22:23:37.661838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.661759 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f770114f-8048-4957-b7cd-704ffe8fafb1" containerName="istio-proxy" Oct 30 22:23:37.661838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.661774 4335 cpu_manager.go:414] "containerMap: removing container" podUID="94191978-6460-43f3-9799-42c7c6286b2a" containerName="payload-processing" Oct 30 22:23:37.661838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.661780 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94191978-6460-43f3-9799-42c7c6286b2a" containerName="payload-processing" Oct 30 22:23:37.661838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.661791 4335 cpu_manager.go:414] "containerMap: removing container" podUID="fcfccb62-83d1-40d8-b0e4-8e04c9319490" containerName="payload-pre-processing" Oct 30 22:23:37.661838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.661795 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcfccb62-83d1-40d8-b0e4-8e04c9319490" containerName="payload-pre-processing" Oct 30 22:23:37.661838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.661816 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5c18511f-79df-44be-807e-f8d2744c7949" containerName="payload-processing" Oct 30 22:23:37.661838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.661821 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c18511f-79df-44be-807e-f8d2744c7949" containerName="payload-processing" Oct 30 22:23:37.661838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.661831 4335 cpu_manager.go:414] "containerMap: removing container" podUID="ef6c24d9-0e05-4181-9c98-dd5015f58975" containerName="istio-proxy" Oct 30 22:23:37.661838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.661836 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6c24d9-0e05-4181-9c98-dd5015f58975" containerName="istio-proxy" Oct 30 22:23:37.662167 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.661948 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="4489c36f-0a7c-4ade-bce8-0ff7b38cec4b" containerName="revoke-keys" Oct 30 22:23:37.662167 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.661958 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f770114f-8048-4957-b7cd-704ffe8fafb1" containerName="istio-proxy" Oct 30 22:23:37.662167 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.661966 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="94191978-6460-43f3-9799-42c7c6286b2a" containerName="payload-processing" Oct 30 22:23:37.662167 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.661972 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcfccb62-83d1-40d8-b0e4-8e04c9319490" containerName="payload-pre-processing" Oct 30 22:23:37.662167 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.661982 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c18511f-79df-44be-807e-f8d2744c7949" containerName="payload-processing" Oct 30 22:23:37.662167 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.661989 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6c24d9-0e05-4181-9c98-dd5015f58975" containerName="istio-proxy" Oct 30 22:23:37.664434 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.664420 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w" Oct 30 22:23:37.701098 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.701067 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e207549a-e5fa-43e5-96e1-45402d3aa40b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w\" (UID: \"e207549a-e5fa-43e5-96e1-45402d3aa40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w" Oct 30 22:23:37.701884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.701860 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e207549a-e5fa-43e5-96e1-45402d3aa40b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w\" (UID: \"e207549a-e5fa-43e5-96e1-45402d3aa40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w" Oct 30 22:23:37.702190 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.702168 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg9bl\" (UniqueName: \"kubernetes.io/projected/e207549a-e5fa-43e5-96e1-45402d3aa40b-kube-api-access-kg9bl\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w\" (UID: \"e207549a-e5fa-43e5-96e1-45402d3aa40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w" Oct 30 22:23:37.742218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.742189 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w"] Oct 30 22:23:37.743601 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.743575 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:23:37.743739 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.743604 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:23:37.784254 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.784226 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:23:37.784254 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.784226 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:23:37.826588 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.826548 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e207549a-e5fa-43e5-96e1-45402d3aa40b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w\" (UID: \"e207549a-e5fa-43e5-96e1-45402d3aa40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w" Oct 30 22:23:37.826898 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.826877 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e207549a-e5fa-43e5-96e1-45402d3aa40b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w\" (UID: \"e207549a-e5fa-43e5-96e1-45402d3aa40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w" Oct 30 22:23:37.827128 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.827107 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg9bl\" (UniqueName: \"kubernetes.io/projected/e207549a-e5fa-43e5-96e1-45402d3aa40b-kube-api-access-kg9bl\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w\" (UID: \"e207549a-e5fa-43e5-96e1-45402d3aa40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w" Oct 30 22:23:37.827928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.827881 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e207549a-e5fa-43e5-96e1-45402d3aa40b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w\" (UID: \"e207549a-e5fa-43e5-96e1-45402d3aa40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w" Oct 30 22:23:37.832053 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.832033 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e207549a-e5fa-43e5-96e1-45402d3aa40b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w\" (UID: \"e207549a-e5fa-43e5-96e1-45402d3aa40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w" Oct 30 22:23:37.889205 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.889184 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg9bl\" (UniqueName: \"kubernetes.io/projected/e207549a-e5fa-43e5-96e1-45402d3aa40b-kube-api-access-kg9bl\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w\" (UID: \"e207549a-e5fa-43e5-96e1-45402d3aa40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w" Oct 30 22:23:37.933177 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.933098 4335 patch_prober.go:29] interesting pod/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 22:23:37.933177 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.933157 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" podUID="4e974fd0-2c92-4626-9673-d202cc596fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 22:23:37.977945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:37.977919 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w" Oct 30 22:23:38.394023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:38.393991 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w"] Oct 30 22:23:38.394294 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:23:38.394275 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode207549a_e5fa_43e5_96e1_45402d3aa40b.slice/crio-611de032d450dc542c511c3676b07aec5c2650bb7b6e8412b7332ee41d4b0aaa WatchSource:0}: Error finding container 611de032d450dc542c511c3676b07aec5c2650bb7b6e8412b7332ee41d4b0aaa: Status 404 returned error can't find the container with id 611de032d450dc542c511c3676b07aec5c2650bb7b6e8412b7332ee41d4b0aaa Oct 30 22:23:38.932435 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:38.932406 4335 patch_prober.go:29] interesting pod/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 22:23:38.932548 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:38.932449 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" podUID="4e974fd0-2c92-4626-9673-d202cc596fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 22:23:38.948532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:38.948503 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w" event={"ID":"e207549a-e5fa-43e5-96e1-45402d3aa40b","Type":"ContainerStarted","Data":"5d6c91c77538092fa9561857932c6ffbc64d3cd8e8e16f3360a7d11520711f4d"} Oct 30 22:23:38.948691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:38.948559 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w" event={"ID":"e207549a-e5fa-43e5-96e1-45402d3aa40b","Type":"ContainerStarted","Data":"611de032d450dc542c511c3676b07aec5c2650bb7b6e8412b7332ee41d4b0aaa"} Oct 30 22:23:39.067329 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.067283 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w" podStartSLOduration=2.067269442 podStartE2EDuration="2.067269442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:39.065059383 +0000 UTC m=+2352.165491114" watchObservedRunningTime="2026-10-30 22:23:39.067269442 +0000 UTC m=+2352.167701174" Oct 30 22:23:39.143744 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.143708 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc"] Oct 30 22:23:39.146935 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.146914 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc" Oct 30 22:23:39.235883 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.235812 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc"] Oct 30 22:23:39.246028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.245990 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc\" (UID: \"af0a2557-34d5-4e1b-8bca-3c459f4fdd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc" Oct 30 22:23:39.246187 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.246077 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc\" (UID: \"af0a2557-34d5-4e1b-8bca-3c459f4fdd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc" Oct 30 22:23:39.246437 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.246412 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5psw\" (UniqueName: \"kubernetes.io/projected/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-kube-api-access-l5psw\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc\" (UID: \"af0a2557-34d5-4e1b-8bca-3c459f4fdd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc" Oct 30 22:23:39.347316 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.347287 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5psw\" (UniqueName: \"kubernetes.io/projected/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-kube-api-access-l5psw\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc\" (UID: \"af0a2557-34d5-4e1b-8bca-3c459f4fdd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc" Oct 30 22:23:39.347479 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.347330 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc\" (UID: \"af0a2557-34d5-4e1b-8bca-3c459f4fdd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc" Oct 30 22:23:39.347479 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.347356 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc\" (UID: \"af0a2557-34d5-4e1b-8bca-3c459f4fdd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc" Oct 30 22:23:39.348019 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.347988 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc\" (UID: \"af0a2557-34d5-4e1b-8bca-3c459f4fdd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc" Oct 30 22:23:39.349800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.349780 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc\" (UID: \"af0a2557-34d5-4e1b-8bca-3c459f4fdd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc" Oct 30 22:23:39.424636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.420995 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5psw\" (UniqueName: \"kubernetes.io/projected/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-kube-api-access-l5psw\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc\" (UID: \"af0a2557-34d5-4e1b-8bca-3c459f4fdd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc" Oct 30 22:23:39.461016 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.460992 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc" Oct 30 22:23:39.876905 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:23:39.876875 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0a2557_34d5_4e1b_8bca_3c459f4fdd52.slice/crio-8b3bde4830e7d83fb711173decad68c0bef7118432753d18f6255891d12daff3 WatchSource:0}: Error finding container 8b3bde4830e7d83fb711173decad68c0bef7118432753d18f6255891d12daff3: Status 404 returned error can't find the container with id 8b3bde4830e7d83fb711173decad68c0bef7118432753d18f6255891d12daff3 Oct 30 22:23:39.877758 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.877731 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc"] Oct 30 22:23:39.932474 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.932446 4335 patch_prober.go:29] interesting pod/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 22:23:39.932608 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.932493 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" podUID="4e974fd0-2c92-4626-9673-d202cc596fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 22:23:39.965204 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.965167 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc" event={"ID":"af0a2557-34d5-4e1b-8bca-3c459f4fdd52","Type":"ContainerStarted","Data":"8b3bde4830e7d83fb711173decad68c0bef7118432753d18f6255891d12daff3"} Oct 30 22:23:39.967061 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.967028 4335 generic.go:356] "Generic (PLEG): container finished" podID="e207549a-e5fa-43e5-96e1-45402d3aa40b" containerID="5d6c91c77538092fa9561857932c6ffbc64d3cd8e8e16f3360a7d11520711f4d" exitCode=0 Oct 30 22:23:39.967189 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:39.967161 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w" event={"ID":"e207549a-e5fa-43e5-96e1-45402d3aa40b","Type":"ContainerDied","Data":"5d6c91c77538092fa9561857932c6ffbc64d3cd8e8e16f3360a7d11520711f4d"} Oct 30 22:23:40.936693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:40.936654 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:23:40.937069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:40.936953 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:23:40.937602 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:40.937586 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:23:40.978402 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:40.978367 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc" event={"ID":"af0a2557-34d5-4e1b-8bca-3c459f4fdd52","Type":"ContainerStarted","Data":"786ca8cf795716b58e62ae1fe50bab3e14e846f1233f2acec48b2fe18bf6d7e4"} Oct 30 22:23:41.150880 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.150857 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w" Oct 30 22:23:41.173731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.173705 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e207549a-e5fa-43e5-96e1-45402d3aa40b-kube-api-access-kg9bl\" (UniqueName: \"kubernetes.io/projected/e207549a-e5fa-43e5-96e1-45402d3aa40b-kube-api-access-kg9bl\") pod \"e207549a-e5fa-43e5-96e1-45402d3aa40b\" (UID: \"e207549a-e5fa-43e5-96e1-45402d3aa40b\") " Oct 30 22:23:41.173868 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.173762 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e207549a-e5fa-43e5-96e1-45402d3aa40b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e207549a-e5fa-43e5-96e1-45402d3aa40b-maas-api-openshift-ca\") pod \"e207549a-e5fa-43e5-96e1-45402d3aa40b\" (UID: \"e207549a-e5fa-43e5-96e1-45402d3aa40b\") " Oct 30 22:23:41.173868 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.173784 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e207549a-e5fa-43e5-96e1-45402d3aa40b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e207549a-e5fa-43e5-96e1-45402d3aa40b-maas-api-xks-ca\") pod \"e207549a-e5fa-43e5-96e1-45402d3aa40b\" (UID: \"e207549a-e5fa-43e5-96e1-45402d3aa40b\") " Oct 30 22:23:41.174172 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.174144 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e207549a-e5fa-43e5-96e1-45402d3aa40b-maas-api-openshift-ca" pod "e207549a-e5fa-43e5-96e1-45402d3aa40b" (UID: "e207549a-e5fa-43e5-96e1-45402d3aa40b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:41.175948 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.175920 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e207549a-e5fa-43e5-96e1-45402d3aa40b-maas-api-xks-ca" pod "e207549a-e5fa-43e5-96e1-45402d3aa40b" (UID: "e207549a-e5fa-43e5-96e1-45402d3aa40b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:41.176043 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.175951 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e207549a-e5fa-43e5-96e1-45402d3aa40b-kube-api-access-kg9bl" pod "e207549a-e5fa-43e5-96e1-45402d3aa40b" (UID: "e207549a-e5fa-43e5-96e1-45402d3aa40b"). InnerVolumeSpecName "kube-api-access-kg9bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:41.275316 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.275284 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg9bl\" (UniqueName: \"kubernetes.io/projected/e207549a-e5fa-43e5-96e1-45402d3aa40b-kube-api-access-kg9bl\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:41.275316 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.275314 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e207549a-e5fa-43e5-96e1-45402d3aa40b-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:41.275552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.275326 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e207549a-e5fa-43e5-96e1-45402d3aa40b-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:41.344907 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.344855 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc" podStartSLOduration=2.344841909 podStartE2EDuration="2.344841909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:41.341892648 +0000 UTC m=+2354.442324381" watchObservedRunningTime="2026-10-30 22:23:41.344841909 +0000 UTC m=+2354.445273692" Oct 30 22:23:41.989319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.989287 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w" Oct 30 22:23:41.989319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.989312 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w" event={"ID":"e207549a-e5fa-43e5-96e1-45402d3aa40b","Type":"ContainerDied","Data":"611de032d450dc542c511c3676b07aec5c2650bb7b6e8412b7332ee41d4b0aaa"} Oct 30 22:23:41.989894 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.989370 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="611de032d450dc542c511c3676b07aec5c2650bb7b6e8412b7332ee41d4b0aaa" Oct 30 22:23:41.992708 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.992685 4335 generic.go:356] "Generic (PLEG): container finished" podID="e7f63a86-2ce4-45ad-8f6a-38025710f1f5" containerID="5642d47bd7500a209cf89862e77d3baee4c4bbc672c30398736b944b1374d51a" exitCode=0 Oct 30 22:23:41.992854 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.992772 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" event={"ID":"e7f63a86-2ce4-45ad-8f6a-38025710f1f5","Type":"ContainerDied","Data":"5642d47bd7500a209cf89862e77d3baee4c4bbc672c30398736b944b1374d51a"} Oct 30 22:23:41.992854 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.992831 4335 scope.go:122] "RemoveContainer" containerID="797a88611d5a92c6fdd8203ced121e2795c14e6c30a5a18512daec383c220d05" Oct 30 22:23:41.994865 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.994845 4335 generic.go:356] "Generic (PLEG): container finished" podID="af0a2557-34d5-4e1b-8bca-3c459f4fdd52" containerID="786ca8cf795716b58e62ae1fe50bab3e14e846f1233f2acec48b2fe18bf6d7e4" exitCode=0 Oct 30 22:23:41.995875 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:41.995855 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc" event={"ID":"af0a2557-34d5-4e1b-8bca-3c459f4fdd52","Type":"ContainerDied","Data":"786ca8cf795716b58e62ae1fe50bab3e14e846f1233f2acec48b2fe18bf6d7e4"} Oct 30 22:23:42.035657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.035639 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:23:42.084834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.084811 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-kube-api-access-jhhsr\" (UniqueName: \"kubernetes.io/projected/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-kube-api-access-jhhsr\") pod \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " Oct 30 22:23:42.084960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.084865 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istiod-ca-cert\") pod \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " Oct 30 22:23:42.084960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.084940 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-workload-certs\") pod \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " Oct 30 22:23:42.085159 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.084998 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-envoy\") pod \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " Oct 30 22:23:42.085159 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.085093 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-podinfo\") pod \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " Oct 30 22:23:42.085159 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.085128 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-workload-socket\") pod \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " Oct 30 22:23:42.085323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.085184 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-credential-socket\") pod \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " Oct 30 22:23:42.085323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.085262 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-token\" (UniqueName: \"kubernetes.io/projected/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-token\") pod \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " Oct 30 22:23:42.085323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.085261 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-workload-certs" pod "e7f63a86-2ce4-45ad-8f6a-38025710f1f5" (UID: "e7f63a86-2ce4-45ad-8f6a-38025710f1f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.085323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.085299 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istiod-ca-cert" pod "e7f63a86-2ce4-45ad-8f6a-38025710f1f5" (UID: "e7f63a86-2ce4-45ad-8f6a-38025710f1f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:42.085554 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.085354 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-data\") pod \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\" (UID: \"e7f63a86-2ce4-45ad-8f6a-38025710f1f5\") " Oct 30 22:23:42.085554 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.085444 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-workload-socket" pod "e7f63a86-2ce4-45ad-8f6a-38025710f1f5" (UID: "e7f63a86-2ce4-45ad-8f6a-38025710f1f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.085721 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.085533 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-credential-socket" pod "e7f63a86-2ce4-45ad-8f6a-38025710f1f5" (UID: "e7f63a86-2ce4-45ad-8f6a-38025710f1f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.085721 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.085677 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-data" pod "e7f63a86-2ce4-45ad-8f6a-38025710f1f5" (UID: "e7f63a86-2ce4-45ad-8f6a-38025710f1f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.086089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.086063 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.086184 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.086090 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.086184 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.086107 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.086184 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.086121 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.086184 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.086135 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.087459 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.087436 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-kube-api-access-jhhsr" pod "e7f63a86-2ce4-45ad-8f6a-38025710f1f5" (UID: "e7f63a86-2ce4-45ad-8f6a-38025710f1f5"). InnerVolumeSpecName "kube-api-access-jhhsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:42.087882 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.087850 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-podinfo" pod "e7f63a86-2ce4-45ad-8f6a-38025710f1f5" (UID: "e7f63a86-2ce4-45ad-8f6a-38025710f1f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:42.087969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.087906 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-token" pod "e7f63a86-2ce4-45ad-8f6a-38025710f1f5" (UID: "e7f63a86-2ce4-45ad-8f6a-38025710f1f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:42.088427 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.088409 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-envoy" pod "e7f63a86-2ce4-45ad-8f6a-38025710f1f5" (UID: "e7f63a86-2ce4-45ad-8f6a-38025710f1f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:42.187960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.187931 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.187960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.187961 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.188140 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.187971 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:42.188140 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:42.187980 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhhsr\" (UniqueName: \"kubernetes.io/projected/e7f63a86-2ce4-45ad-8f6a-38025710f1f5-kube-api-access-jhhsr\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:43.005338 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:43.005305 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" Oct 30 22:23:43.005338 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:43.005315 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p" event={"ID":"e7f63a86-2ce4-45ad-8f6a-38025710f1f5","Type":"ContainerDied","Data":"142b9de95e0ad30b461917333fcc4e1f66856845cb77ba2d0ce6aea7fc862418"} Oct 30 22:23:43.005967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:43.005376 4335 scope.go:122] "RemoveContainer" containerID="5642d47bd7500a209cf89862e77d3baee4c4bbc672c30398736b944b1374d51a" Oct 30 22:23:43.168747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:43.168723 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc" Oct 30 22:23:43.172979 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:43.172946 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p"] Oct 30 22:23:43.199442 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:43.199418 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-maas-api-openshift-ca\") pod \"af0a2557-34d5-4e1b-8bca-3c459f4fdd52\" (UID: \"af0a2557-34d5-4e1b-8bca-3c459f4fdd52\") " Oct 30 22:23:43.199558 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:43.199537 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-maas-api-xks-ca\") pod \"af0a2557-34d5-4e1b-8bca-3c459f4fdd52\" (UID: \"af0a2557-34d5-4e1b-8bca-3c459f4fdd52\") " Oct 30 22:23:43.199810 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:43.199789 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-kube-api-access-l5psw\" (UniqueName: \"kubernetes.io/projected/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-kube-api-access-l5psw\") pod \"af0a2557-34d5-4e1b-8bca-3c459f4fdd52\" (UID: \"af0a2557-34d5-4e1b-8bca-3c459f4fdd52\") " Oct 30 22:23:43.199906 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:43.199817 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-maas-api-openshift-ca" pod "af0a2557-34d5-4e1b-8bca-3c459f4fdd52" (UID: "af0a2557-34d5-4e1b-8bca-3c459f4fdd52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:43.200259 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:43.200240 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:43.201784 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:43.201764 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-maas-api-xks-ca" pod "af0a2557-34d5-4e1b-8bca-3c459f4fdd52" (UID: "af0a2557-34d5-4e1b-8bca-3c459f4fdd52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:43.201967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:43.201943 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-kube-api-access-l5psw" pod "af0a2557-34d5-4e1b-8bca-3c459f4fdd52" (UID: "af0a2557-34d5-4e1b-8bca-3c459f4fdd52"). InnerVolumeSpecName "kube-api-access-l5psw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:43.221998 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:43.221974 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ec058c-openshift-default-664c75c54c-sws5p"] Oct 30 22:23:43.301750 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:43.301725 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:43.301871 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:43.301752 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5psw\" (UniqueName: \"kubernetes.io/projected/af0a2557-34d5-4e1b-8bca-3c459f4fdd52-kube-api-access-l5psw\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:44.016757 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:44.016720 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc" Oct 30 22:23:44.016757 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:44.016750 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc" event={"ID":"af0a2557-34d5-4e1b-8bca-3c459f4fdd52","Type":"ContainerDied","Data":"8b3bde4830e7d83fb711173decad68c0bef7118432753d18f6255891d12daff3"} Oct 30 22:23:44.017247 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:44.016831 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b3bde4830e7d83fb711173decad68c0bef7118432753d18f6255891d12daff3" Oct 30 22:23:45.060426 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:45.060392 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f63a86-2ce4-45ad-8f6a-38025710f1f5" path="/var/lib/kubelet/pods/e7f63a86-2ce4-45ad-8f6a-38025710f1f5/volumes" Oct 30 22:23:47.941922 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:47.941883 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd"] Oct 30 22:23:50.315640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:50.315248 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22"] Oct 30 22:23:52.943352 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:52.943278 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd" podUID="591e6aee-d698-4ced-9c55-0e0db3697bde" containerName="payload-pre-processing" containerID="cri-o://f55da982726289370af5da2b2009e05ed8ea741282f0f761f995111ccce7f454" gracePeriod=25 Oct 30 22:23:53.102361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:53.101972 4335 generic.go:356] "Generic (PLEG): container finished" podID="591e6aee-d698-4ced-9c55-0e0db3697bde" containerID="f55da982726289370af5da2b2009e05ed8ea741282f0f761f995111ccce7f454" exitCode=0 Oct 30 22:23:53.102361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:53.102277 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd" event={"ID":"591e6aee-d698-4ced-9c55-0e0db3697bde","Type":"ContainerDied","Data":"f55da982726289370af5da2b2009e05ed8ea741282f0f761f995111ccce7f454"} Oct 30 22:23:53.243071 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:53.242703 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd" Oct 30 22:23:53.395942 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:53.395909 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591e6aee-d698-4ced-9c55-0e0db3697bde-kube-api-access-h5m52\" (UniqueName: \"kubernetes.io/projected/591e6aee-d698-4ced-9c55-0e0db3697bde-kube-api-access-h5m52\") pod \"591e6aee-d698-4ced-9c55-0e0db3697bde\" (UID: \"591e6aee-d698-4ced-9c55-0e0db3697bde\") " Oct 30 22:23:53.396119 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:53.396020 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/591e6aee-d698-4ced-9c55-0e0db3697bde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591e6aee-d698-4ced-9c55-0e0db3697bde-plugins-config-volume\") pod \"591e6aee-d698-4ced-9c55-0e0db3697bde\" (UID: \"591e6aee-d698-4ced-9c55-0e0db3697bde\") " Oct 30 22:23:53.396653 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:53.396598 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591e6aee-d698-4ced-9c55-0e0db3697bde-plugins-config-volume" pod "591e6aee-d698-4ced-9c55-0e0db3697bde" (UID: "591e6aee-d698-4ced-9c55-0e0db3697bde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:53.398967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:53.398938 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591e6aee-d698-4ced-9c55-0e0db3697bde-kube-api-access-h5m52" pod "591e6aee-d698-4ced-9c55-0e0db3697bde" (UID: "591e6aee-d698-4ced-9c55-0e0db3697bde"). InnerVolumeSpecName "kube-api-access-h5m52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:53.498302 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:53.498224 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5m52\" (UniqueName: \"kubernetes.io/projected/591e6aee-d698-4ced-9c55-0e0db3697bde-kube-api-access-h5m52\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:53.498302 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:53.498257 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591e6aee-d698-4ced-9c55-0e0db3697bde-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:54.118636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:54.118592 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd" Oct 30 22:23:54.119703 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:54.118590 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd" event={"ID":"591e6aee-d698-4ced-9c55-0e0db3697bde","Type":"ContainerDied","Data":"d0d536381d256124a692e8081f7b4f6b5dcf9d5c3f79f04d3ca39b5e5ef72b66"} Oct 30 22:23:54.119928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:54.119911 4335 scope.go:122] "RemoveContainer" containerID="f55da982726289370af5da2b2009e05ed8ea741282f0f761f995111ccce7f454" Oct 30 22:23:54.287081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:54.287042 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd"] Oct 30 22:23:54.338151 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:54.338108 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f3344-79f96fbb59-n5cfd"] Oct 30 22:23:55.059969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:55.059930 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="591e6aee-d698-4ced-9c55-0e0db3697bde" path="/var/lib/kubelet/pods/591e6aee-d698-4ced-9c55-0e0db3697bde/volumes" Oct 30 22:23:55.316711 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:55.316137 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22" podUID="171de552-11b5-4865-b021-92ae8807842d" containerName="payload-processing" containerID="cri-o://8b9fdf15aa4d9d40b94cea0b9a88971a8008b45959e1a793b24246e542d85cbf" gracePeriod=25 Oct 30 22:23:55.343033 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:23:55.342863 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:23:55.633468 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:55.633438 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22" Oct 30 22:23:55.729946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:55.729917 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/171de552-11b5-4865-b021-92ae8807842d-kube-api-access-g6c6f\" (UniqueName: \"kubernetes.io/projected/171de552-11b5-4865-b021-92ae8807842d-kube-api-access-g6c6f\") pod \"171de552-11b5-4865-b021-92ae8807842d\" (UID: \"171de552-11b5-4865-b021-92ae8807842d\") " Oct 30 22:23:55.730135 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:55.730048 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/171de552-11b5-4865-b021-92ae8807842d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171de552-11b5-4865-b021-92ae8807842d-plugins-config-volume\") pod \"171de552-11b5-4865-b021-92ae8807842d\" (UID: \"171de552-11b5-4865-b021-92ae8807842d\") " Oct 30 22:23:55.730496 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:55.730472 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/171de552-11b5-4865-b021-92ae8807842d-plugins-config-volume" pod "171de552-11b5-4865-b021-92ae8807842d" (UID: "171de552-11b5-4865-b021-92ae8807842d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:55.732374 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:55.732345 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171de552-11b5-4865-b021-92ae8807842d-kube-api-access-g6c6f" pod "171de552-11b5-4865-b021-92ae8807842d" (UID: "171de552-11b5-4865-b021-92ae8807842d"). InnerVolumeSpecName "kube-api-access-g6c6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:55.837091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:55.837023 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6c6f\" (UniqueName: \"kubernetes.io/projected/171de552-11b5-4865-b021-92ae8807842d-kube-api-access-g6c6f\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:55.837091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:55.837062 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171de552-11b5-4865-b021-92ae8807842d-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:56.145032 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:56.145001 4335 generic.go:356] "Generic (PLEG): container finished" podID="171de552-11b5-4865-b021-92ae8807842d" containerID="8b9fdf15aa4d9d40b94cea0b9a88971a8008b45959e1a793b24246e542d85cbf" exitCode=0 Oct 30 22:23:56.145221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:56.145049 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22" Oct 30 22:23:56.145221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:56.145093 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22" event={"ID":"171de552-11b5-4865-b021-92ae8807842d","Type":"ContainerDied","Data":"8b9fdf15aa4d9d40b94cea0b9a88971a8008b45959e1a793b24246e542d85cbf"} Oct 30 22:23:56.145221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:56.145155 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22" event={"ID":"171de552-11b5-4865-b021-92ae8807842d","Type":"ContainerDied","Data":"0ab5115a3fd5487eed2fb4d638d121536786341986a5c7572908ffb275bceeb7"} Oct 30 22:23:56.145221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:56.145184 4335 scope.go:122] "RemoveContainer" containerID="8b9fdf15aa4d9d40b94cea0b9a88971a8008b45959e1a793b24246e542d85cbf" Oct 30 22:23:56.159673 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:56.159657 4335 scope.go:122] "RemoveContainer" containerID="8b9fdf15aa4d9d40b94cea0b9a88971a8008b45959e1a793b24246e542d85cbf" Oct 30 22:23:56.159936 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:23:56.159919 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9fdf15aa4d9d40b94cea0b9a88971a8008b45959e1a793b24246e542d85cbf\": container with ID starting with 8b9fdf15aa4d9d40b94cea0b9a88971a8008b45959e1a793b24246e542d85cbf not found: ID does not exist" containerID="8b9fdf15aa4d9d40b94cea0b9a88971a8008b45959e1a793b24246e542d85cbf" Oct 30 22:23:56.159986 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:56.159943 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9fdf15aa4d9d40b94cea0b9a88971a8008b45959e1a793b24246e542d85cbf"} err="failed to get container status \"8b9fdf15aa4d9d40b94cea0b9a88971a8008b45959e1a793b24246e542d85cbf\": rpc error: code = NotFound desc = could not find container \"8b9fdf15aa4d9d40b94cea0b9a88971a8008b45959e1a793b24246e542d85cbf\": container with ID starting with 8b9fdf15aa4d9d40b94cea0b9a88971a8008b45959e1a793b24246e542d85cbf not found: ID does not exist" Oct 30 22:23:56.319939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:56.319904 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22"] Oct 30 22:23:56.365457 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:56.365426 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-9a96b4-c8d59c7b9-r6k22"] Oct 30 22:23:57.060430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:57.060404 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="171de552-11b5-4865-b021-92ae8807842d" path="/var/lib/kubelet/pods/171de552-11b5-4865-b021-92ae8807842d/volumes" Oct 30 22:23:57.155804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:57.155779 4335 generic.go:356] "Generic (PLEG): container finished" podID="fc259675-c041-4deb-aaf3-ac3c90c48a05" containerID="6e9b65a2165157264b6ca29415de1e43815cc5a204a0cdcdcc1a7ba2f0ed62f8" exitCode=0 Oct 30 22:23:57.155941 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:57.155855 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" event={"ID":"fc259675-c041-4deb-aaf3-ac3c90c48a05","Type":"ContainerDied","Data":"6e9b65a2165157264b6ca29415de1e43815cc5a204a0cdcdcc1a7ba2f0ed62f8"} Oct 30 22:23:57.156210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:57.156196 4335 scope.go:122] "RemoveContainer" containerID="6e9b65a2165157264b6ca29415de1e43815cc5a204a0cdcdcc1a7ba2f0ed62f8" Oct 30 22:23:57.157793 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:57.157766 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:23:57.157908 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:57.157838 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:23:57.157908 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:57.157887 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:23:57.478264 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:57.478235 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:23:58.166995 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:58.166964 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" event={"ID":"fc259675-c041-4deb-aaf3-ac3c90c48a05","Type":"ContainerStarted","Data":"ac2da5b43441976bfaba61c971f747b569c528b5488e657d94b37a75f4d27e84"} Oct 30 22:23:58.963163 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:58.963126 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:23:58.964654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:58.964601 4335 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:23:58.964814 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:58.964690 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" podUID="fc259675-c041-4deb-aaf3-ac3c90c48a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 22:23:59.963148 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:59.963116 4335 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:23:59.963351 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:23:59.963166 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" podUID="fc259675-c041-4deb-aaf3-ac3c90c48a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 22:24:00.963179 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:00.963138 4335 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:24:00.963402 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:00.963199 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" podUID="fc259675-c041-4deb-aaf3-ac3c90c48a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 22:24:01.963154 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:01.963116 4335 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:24:01.963332 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:01.963180 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" podUID="fc259675-c041-4deb-aaf3-ac3c90c48a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 22:24:02.963163 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:02.963131 4335 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:24:02.963546 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:02.963185 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" podUID="fc259675-c041-4deb-aaf3-ac3c90c48a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 22:24:03.968271 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:03.968243 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:24:03.968802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:03.968598 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:24:03.969209 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:03.969193 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:24:04.940388 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.940355 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r"] Oct 30 22:24:04.941048 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.941034 4335 cpu_manager.go:414] "containerMap: removing container" podUID="af0a2557-34d5-4e1b-8bca-3c459f4fdd52" containerName="revoke-keys" Oct 30 22:24:04.941112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.941049 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0a2557-34d5-4e1b-8bca-3c459f4fdd52" containerName="revoke-keys" Oct 30 22:24:04.941112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.941066 4335 cpu_manager.go:414] "containerMap: removing container" podUID="e207549a-e5fa-43e5-96e1-45402d3aa40b" containerName="revoke-keys" Oct 30 22:24:04.941112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.941071 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e207549a-e5fa-43e5-96e1-45402d3aa40b" containerName="revoke-keys" Oct 30 22:24:04.941112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.941086 4335 cpu_manager.go:414] "containerMap: removing container" podUID="e7f63a86-2ce4-45ad-8f6a-38025710f1f5" containerName="istio-proxy" Oct 30 22:24:04.941112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.941092 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f63a86-2ce4-45ad-8f6a-38025710f1f5" containerName="istio-proxy" Oct 30 22:24:04.941112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.941108 4335 cpu_manager.go:414] "containerMap: removing container" podUID="591e6aee-d698-4ced-9c55-0e0db3697bde" containerName="payload-pre-processing" Oct 30 22:24:04.941112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.941112 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="591e6aee-d698-4ced-9c55-0e0db3697bde" containerName="payload-pre-processing" Oct 30 22:24:04.941371 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.941133 4335 cpu_manager.go:414] "containerMap: removing container" podUID="171de552-11b5-4865-b021-92ae8807842d" containerName="payload-processing" Oct 30 22:24:04.941371 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.941137 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="171de552-11b5-4865-b021-92ae8807842d" containerName="payload-processing" Oct 30 22:24:04.941371 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.941242 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f63a86-2ce4-45ad-8f6a-38025710f1f5" containerName="istio-proxy" Oct 30 22:24:04.941371 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.941251 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="591e6aee-d698-4ced-9c55-0e0db3697bde" containerName="payload-pre-processing" Oct 30 22:24:04.941371 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.941261 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="171de552-11b5-4865-b021-92ae8807842d" containerName="payload-processing" Oct 30 22:24:04.941371 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.941269 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f63a86-2ce4-45ad-8f6a-38025710f1f5" containerName="istio-proxy" Oct 30 22:24:04.941371 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.941277 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0a2557-34d5-4e1b-8bca-3c459f4fdd52" containerName="revoke-keys" Oct 30 22:24:04.941371 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.941284 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="e207549a-e5fa-43e5-96e1-45402d3aa40b" containerName="revoke-keys" Oct 30 22:24:04.945027 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.945008 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r" Oct 30 22:24:04.970861 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.970834 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11603f83-ff6f-421c-92a7-dd29e0721270-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e3324355-8n99r\" (UID: \"11603f83-ff6f-421c-92a7-dd29e0721270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r" Oct 30 22:24:04.971228 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.970881 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11603f83-ff6f-421c-92a7-dd29e0721270-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e3324355-8n99r\" (UID: \"11603f83-ff6f-421c-92a7-dd29e0721270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r" Oct 30 22:24:04.971228 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.970913 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prv68\" (UniqueName: \"kubernetes.io/projected/11603f83-ff6f-421c-92a7-dd29e0721270-kube-api-access-prv68\") pod \"maas-api-revoke-keys-e2e-mt-e3324355-8n99r\" (UID: \"11603f83-ff6f-421c-92a7-dd29e0721270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r" Oct 30 22:24:04.997712 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.997691 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:24:04.997926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.997818 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:24:04.998261 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:04.998247 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:24:05.003947 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:05.003921 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r"] Oct 30 22:24:05.045500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:05.045473 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:24:05.071898 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:05.071869 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11603f83-ff6f-421c-92a7-dd29e0721270-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e3324355-8n99r\" (UID: \"11603f83-ff6f-421c-92a7-dd29e0721270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r" Oct 30 22:24:05.072034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:05.071927 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11603f83-ff6f-421c-92a7-dd29e0721270-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e3324355-8n99r\" (UID: \"11603f83-ff6f-421c-92a7-dd29e0721270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r" Oct 30 22:24:05.072034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:05.071974 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prv68\" (UniqueName: \"kubernetes.io/projected/11603f83-ff6f-421c-92a7-dd29e0721270-kube-api-access-prv68\") pod \"maas-api-revoke-keys-e2e-mt-e3324355-8n99r\" (UID: \"11603f83-ff6f-421c-92a7-dd29e0721270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r" Oct 30 22:24:05.072654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:05.072609 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11603f83-ff6f-421c-92a7-dd29e0721270-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e3324355-8n99r\" (UID: \"11603f83-ff6f-421c-92a7-dd29e0721270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r" Oct 30 22:24:05.074592 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:05.074568 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11603f83-ff6f-421c-92a7-dd29e0721270-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e3324355-8n99r\" (UID: \"11603f83-ff6f-421c-92a7-dd29e0721270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r" Oct 30 22:24:05.133581 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:05.133554 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prv68\" (UniqueName: \"kubernetes.io/projected/11603f83-ff6f-421c-92a7-dd29e0721270-kube-api-access-prv68\") pod \"maas-api-revoke-keys-e2e-mt-e3324355-8n99r\" (UID: \"11603f83-ff6f-421c-92a7-dd29e0721270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r" Oct 30 22:24:05.263061 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:05.262958 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r" Oct 30 22:24:05.621109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:05.621077 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:24:05.737814 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:24:05.737783 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11603f83_ff6f_421c_92a7_dd29e0721270.slice/crio-986b6119128231454fbef818b7f66d73f31ce93652c59f42943acb0a6c775c48 WatchSource:0}: Error finding container 986b6119128231454fbef818b7f66d73f31ce93652c59f42943acb0a6c775c48: Status 404 returned error can't find the container with id 986b6119128231454fbef818b7f66d73f31ce93652c59f42943acb0a6c775c48 Oct 30 22:24:05.738495 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:05.738389 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r"] Oct 30 22:24:06.290582 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:06.290509 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r" event={"ID":"11603f83-ff6f-421c-92a7-dd29e0721270","Type":"ContainerStarted","Data":"f9c9002c6e87715ba2b26115c4080ef2b2715381e1bdeef8be962d89afda461b"} Oct 30 22:24:06.290582 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:06.290562 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r" event={"ID":"11603f83-ff6f-421c-92a7-dd29e0721270","Type":"ContainerStarted","Data":"986b6119128231454fbef818b7f66d73f31ce93652c59f42943acb0a6c775c48"} Oct 30 22:24:06.407587 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:06.407538 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r" podStartSLOduration=2.407525262 podStartE2EDuration="2.407525262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:06.405154207 +0000 UTC m=+2379.505585938" watchObservedRunningTime="2026-10-30 22:24:06.407525262 +0000 UTC m=+2379.507956994" Oct 30 22:24:07.323013 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:07.322964 4335 generic.go:356] "Generic (PLEG): container finished" podID="11603f83-ff6f-421c-92a7-dd29e0721270" containerID="f9c9002c6e87715ba2b26115c4080ef2b2715381e1bdeef8be962d89afda461b" exitCode=0 Oct 30 22:24:07.323419 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:07.323103 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r" event={"ID":"11603f83-ff6f-421c-92a7-dd29e0721270","Type":"ContainerDied","Data":"f9c9002c6e87715ba2b26115c4080ef2b2715381e1bdeef8be962d89afda461b"} Oct 30 22:24:08.529217 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:08.529192 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r" Oct 30 22:24:08.536579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:08.536556 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11603f83-ff6f-421c-92a7-dd29e0721270-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11603f83-ff6f-421c-92a7-dd29e0721270-maas-api-openshift-ca\") pod \"11603f83-ff6f-421c-92a7-dd29e0721270\" (UID: \"11603f83-ff6f-421c-92a7-dd29e0721270\") " Oct 30 22:24:08.536744 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:08.536651 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11603f83-ff6f-421c-92a7-dd29e0721270-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11603f83-ff6f-421c-92a7-dd29e0721270-maas-api-xks-ca\") pod \"11603f83-ff6f-421c-92a7-dd29e0721270\" (UID: \"11603f83-ff6f-421c-92a7-dd29e0721270\") " Oct 30 22:24:08.536744 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:08.536741 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11603f83-ff6f-421c-92a7-dd29e0721270-kube-api-access-prv68\" (UniqueName: \"kubernetes.io/projected/11603f83-ff6f-421c-92a7-dd29e0721270-kube-api-access-prv68\") pod \"11603f83-ff6f-421c-92a7-dd29e0721270\" (UID: \"11603f83-ff6f-421c-92a7-dd29e0721270\") " Oct 30 22:24:08.536975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:08.536949 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11603f83-ff6f-421c-92a7-dd29e0721270-maas-api-openshift-ca" pod "11603f83-ff6f-421c-92a7-dd29e0721270" (UID: "11603f83-ff6f-421c-92a7-dd29e0721270"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:08.537347 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:08.537319 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11603f83-ff6f-421c-92a7-dd29e0721270-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:08.539038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:08.538991 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11603f83-ff6f-421c-92a7-dd29e0721270-kube-api-access-prv68" pod "11603f83-ff6f-421c-92a7-dd29e0721270" (UID: "11603f83-ff6f-421c-92a7-dd29e0721270"). InnerVolumeSpecName "kube-api-access-prv68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:08.539238 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:08.539215 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11603f83-ff6f-421c-92a7-dd29e0721270-maas-api-xks-ca" pod "11603f83-ff6f-421c-92a7-dd29e0721270" (UID: "11603f83-ff6f-421c-92a7-dd29e0721270"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:08.638731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:08.638662 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11603f83-ff6f-421c-92a7-dd29e0721270-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:08.638731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:08.638687 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prv68\" (UniqueName: \"kubernetes.io/projected/11603f83-ff6f-421c-92a7-dd29e0721270-kube-api-access-prv68\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:09.340895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:09.340802 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r" event={"ID":"11603f83-ff6f-421c-92a7-dd29e0721270","Type":"ContainerDied","Data":"986b6119128231454fbef818b7f66d73f31ce93652c59f42943acb0a6c775c48"} Oct 30 22:24:09.340895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:09.340844 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r" Oct 30 22:24:09.341125 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:09.340851 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="986b6119128231454fbef818b7f66d73f31ce93652c59f42943acb0a6c775c48" Oct 30 22:24:11.290181 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:11.290145 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5"] Oct 30 22:24:11.338723 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:11.338691 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc5fe-4jxw5"] Oct 30 22:24:13.060749 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:13.060723 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c029ed3-d7c8-4a30-8b3b-b60266c70378" path="/var/lib/kubelet/pods/2c029ed3-d7c8-4a30-8b3b-b60266c70378/volumes" Oct 30 22:24:15.373352 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:15.373316 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl"] Oct 30 22:24:15.480445 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:15.480414 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7"] Oct 30 22:24:17.413934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:17.413894 4335 generic.go:356] "Generic (PLEG): container finished" podID="58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" containerID="fb0c89f770165718e486203b6306cfe225cf43f9c91266f30e9e15e5dfbbe74f" exitCode=0 Oct 30 22:24:17.414427 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:17.413973 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" event={"ID":"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a","Type":"ContainerDied","Data":"fb0c89f770165718e486203b6306cfe225cf43f9c91266f30e9e15e5dfbbe74f"} Oct 30 22:24:17.414427 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:17.414417 4335 scope.go:122] "RemoveContainer" containerID="fb0c89f770165718e486203b6306cfe225cf43f9c91266f30e9e15e5dfbbe74f" Oct 30 22:24:17.416280 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:17.416251 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:24:17.416373 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:17.416324 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:24:17.416419 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:17.416378 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:24:17.815156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:17.815127 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc"] Oct 30 22:24:17.819170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:17.819141 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9a96b4-k92wc"] Oct 30 22:24:18.436822 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:18.436783 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" event={"ID":"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a","Type":"ContainerStarted","Data":"9b99750cf13046cc3352d003c8e66342d562e720518a4e3f378c931820fec202"} Oct 30 22:24:18.591501 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:18.591470 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:24:18.592679 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:18.592641 4335 patch_prober.go:29] interesting pod/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 30 22:24:18.592848 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:18.592721 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" podUID="58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 30 22:24:19.060303 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:19.060272 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af0a2557-34d5-4e1b-8bca-3c459f4fdd52" path="/var/lib/kubelet/pods/af0a2557-34d5-4e1b-8bca-3c459f4fdd52/volumes" Oct 30 22:24:19.586227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:19.586194 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:24:19.591559 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:19.591535 4335 patch_prober.go:29] interesting pod/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 30 22:24:19.591679 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:19.591594 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" podUID="58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 30 22:24:20.373881 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.373822 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl" podUID="169efb02-ef71-4be0-be80-ca561af3d681" containerName="payload-processing" containerID="cri-o://459edf88a0b29de4992f041f53aa6bf172fdf5e3b29bdac7b4195da4e0336c70" gracePeriod=25 Oct 30 22:24:20.481017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.480983 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7" podUID="770e0179-e20c-40ab-83f7-b8367aa0afb1" containerName="payload-pre-processing" containerID="cri-o://5e251bf9381cf7eec7b5f05e2a570468c2673393623560121ae83c1d63c54483" gracePeriod=25 Oct 30 22:24:20.617640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.592377 4335 patch_prober.go:29] interesting pod/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 30 22:24:20.617640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.592442 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" podUID="58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 30 22:24:20.687949 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.687923 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl" Oct 30 22:24:20.847116 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.847074 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169efb02-ef71-4be0-be80-ca561af3d681-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169efb02-ef71-4be0-be80-ca561af3d681-plugins-config-volume\") pod \"169efb02-ef71-4be0-be80-ca561af3d681\" (UID: \"169efb02-ef71-4be0-be80-ca561af3d681\") " Oct 30 22:24:20.847862 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.847730 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169efb02-ef71-4be0-be80-ca561af3d681-plugins-config-volume" pod "169efb02-ef71-4be0-be80-ca561af3d681" (UID: "169efb02-ef71-4be0-be80-ca561af3d681"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:20.848166 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.848150 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169efb02-ef71-4be0-be80-ca561af3d681-kube-api-access-rzd7w\" (UniqueName: \"kubernetes.io/projected/169efb02-ef71-4be0-be80-ca561af3d681-kube-api-access-rzd7w\") pod \"169efb02-ef71-4be0-be80-ca561af3d681\" (UID: \"169efb02-ef71-4be0-be80-ca561af3d681\") " Oct 30 22:24:20.848953 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.848929 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169efb02-ef71-4be0-be80-ca561af3d681-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:20.861684 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.861287 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169efb02-ef71-4be0-be80-ca561af3d681-kube-api-access-rzd7w" pod "169efb02-ef71-4be0-be80-ca561af3d681" (UID: "169efb02-ef71-4be0-be80-ca561af3d681"). InnerVolumeSpecName "kube-api-access-rzd7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:20.877978 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.877923 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7" Oct 30 22:24:20.950237 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.950202 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/770e0179-e20c-40ab-83f7-b8367aa0afb1-kube-api-access-ng7gt\" (UniqueName: \"kubernetes.io/projected/770e0179-e20c-40ab-83f7-b8367aa0afb1-kube-api-access-ng7gt\") pod \"770e0179-e20c-40ab-83f7-b8367aa0afb1\" (UID: \"770e0179-e20c-40ab-83f7-b8367aa0afb1\") " Oct 30 22:24:20.950443 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.950266 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/770e0179-e20c-40ab-83f7-b8367aa0afb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770e0179-e20c-40ab-83f7-b8367aa0afb1-plugins-config-volume\") pod \"770e0179-e20c-40ab-83f7-b8367aa0afb1\" (UID: \"770e0179-e20c-40ab-83f7-b8367aa0afb1\") " Oct 30 22:24:20.950666 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.950640 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770e0179-e20c-40ab-83f7-b8367aa0afb1-plugins-config-volume" pod "770e0179-e20c-40ab-83f7-b8367aa0afb1" (UID: "770e0179-e20c-40ab-83f7-b8367aa0afb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:20.950863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.950846 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzd7w\" (UniqueName: \"kubernetes.io/projected/169efb02-ef71-4be0-be80-ca561af3d681-kube-api-access-rzd7w\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:20.950905 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.950870 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770e0179-e20c-40ab-83f7-b8367aa0afb1-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:20.952478 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.952460 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770e0179-e20c-40ab-83f7-b8367aa0afb1-kube-api-access-ng7gt" pod "770e0179-e20c-40ab-83f7-b8367aa0afb1" (UID: "770e0179-e20c-40ab-83f7-b8367aa0afb1"). InnerVolumeSpecName "kube-api-access-ng7gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:20.989649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.989605 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9"] Oct 30 22:24:20.990318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.990303 4335 cpu_manager.go:414] "containerMap: removing container" podUID="11603f83-ff6f-421c-92a7-dd29e0721270" containerName="revoke-keys" Oct 30 22:24:20.990365 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.990320 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11603f83-ff6f-421c-92a7-dd29e0721270" containerName="revoke-keys" Oct 30 22:24:20.990365 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.990334 4335 cpu_manager.go:414] "containerMap: removing container" podUID="e7f63a86-2ce4-45ad-8f6a-38025710f1f5" containerName="istio-proxy" Oct 30 22:24:20.990365 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.990339 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f63a86-2ce4-45ad-8f6a-38025710f1f5" containerName="istio-proxy" Oct 30 22:24:20.990365 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.990357 4335 cpu_manager.go:414] "containerMap: removing container" podUID="770e0179-e20c-40ab-83f7-b8367aa0afb1" containerName="payload-pre-processing" Oct 30 22:24:20.990365 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.990362 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="770e0179-e20c-40ab-83f7-b8367aa0afb1" containerName="payload-pre-processing" Oct 30 22:24:20.990524 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.990378 4335 cpu_manager.go:414] "containerMap: removing container" podUID="169efb02-ef71-4be0-be80-ca561af3d681" containerName="payload-processing" Oct 30 22:24:20.990524 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.990383 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169efb02-ef71-4be0-be80-ca561af3d681" containerName="payload-processing" Oct 30 22:24:20.990524 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.990479 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="11603f83-ff6f-421c-92a7-dd29e0721270" containerName="revoke-keys" Oct 30 22:24:20.990524 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.990490 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="770e0179-e20c-40ab-83f7-b8367aa0afb1" containerName="payload-pre-processing" Oct 30 22:24:20.990524 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.990498 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="169efb02-ef71-4be0-be80-ca561af3d681" containerName="payload-processing" Oct 30 22:24:20.992959 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:20.992939 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9" Oct 30 22:24:21.052753 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.049128 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:24:21.052753 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.049253 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:24:21.066369 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.066291 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng7gt\" (UniqueName: \"kubernetes.io/projected/770e0179-e20c-40ab-83f7-b8367aa0afb1-kube-api-access-ng7gt\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:21.066671 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.066647 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:24:21.066892 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.066760 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:24:21.070989 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.070966 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9"] Oct 30 22:24:21.167139 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.167059 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdftj\" (UniqueName: \"kubernetes.io/projected/4dd15165-4075-470b-92ab-8246e2fc26b4-kube-api-access-gdftj\") pod \"maas-api-revoke-keys-e2e-api-a-a1024a-z54t9\" (UID: \"4dd15165-4075-470b-92ab-8246e2fc26b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9" Oct 30 22:24:21.167139 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.167107 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd15165-4075-470b-92ab-8246e2fc26b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1024a-z54t9\" (UID: \"4dd15165-4075-470b-92ab-8246e2fc26b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9" Oct 30 22:24:21.167370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.167331 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd15165-4075-470b-92ab-8246e2fc26b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1024a-z54t9\" (UID: \"4dd15165-4075-470b-92ab-8246e2fc26b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9" Oct 30 22:24:21.268930 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.268899 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd15165-4075-470b-92ab-8246e2fc26b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1024a-z54t9\" (UID: \"4dd15165-4075-470b-92ab-8246e2fc26b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9" Oct 30 22:24:21.269110 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.269018 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdftj\" (UniqueName: \"kubernetes.io/projected/4dd15165-4075-470b-92ab-8246e2fc26b4-kube-api-access-gdftj\") pod \"maas-api-revoke-keys-e2e-api-a-a1024a-z54t9\" (UID: \"4dd15165-4075-470b-92ab-8246e2fc26b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9" Oct 30 22:24:21.269110 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.269045 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd15165-4075-470b-92ab-8246e2fc26b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1024a-z54t9\" (UID: \"4dd15165-4075-470b-92ab-8246e2fc26b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9" Oct 30 22:24:21.269543 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.269515 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd15165-4075-470b-92ab-8246e2fc26b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1024a-z54t9\" (UID: \"4dd15165-4075-470b-92ab-8246e2fc26b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9" Oct 30 22:24:21.271509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.271483 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd15165-4075-470b-92ab-8246e2fc26b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1024a-z54t9\" (UID: \"4dd15165-4075-470b-92ab-8246e2fc26b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9" Oct 30 22:24:21.330257 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.330228 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdftj\" (UniqueName: \"kubernetes.io/projected/4dd15165-4075-470b-92ab-8246e2fc26b4-kube-api-access-gdftj\") pod \"maas-api-revoke-keys-e2e-api-a-a1024a-z54t9\" (UID: \"4dd15165-4075-470b-92ab-8246e2fc26b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9" Oct 30 22:24:21.463431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.463343 4335 generic.go:356] "Generic (PLEG): container finished" podID="770e0179-e20c-40ab-83f7-b8367aa0afb1" containerID="5e251bf9381cf7eec7b5f05e2a570468c2673393623560121ae83c1d63c54483" exitCode=0 Oct 30 22:24:21.463431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.463409 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7" Oct 30 22:24:21.463677 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.463435 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7" event={"ID":"770e0179-e20c-40ab-83f7-b8367aa0afb1","Type":"ContainerDied","Data":"5e251bf9381cf7eec7b5f05e2a570468c2673393623560121ae83c1d63c54483"} Oct 30 22:24:21.463677 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.463489 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7" event={"ID":"770e0179-e20c-40ab-83f7-b8367aa0afb1","Type":"ContainerDied","Data":"caab6212cba016a750009dbbe91d05874aed29950e332bed2b218ff23f95f8d8"} Oct 30 22:24:21.463677 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.463511 4335 scope.go:122] "RemoveContainer" containerID="5e251bf9381cf7eec7b5f05e2a570468c2673393623560121ae83c1d63c54483" Oct 30 22:24:21.465190 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.465170 4335 generic.go:356] "Generic (PLEG): container finished" podID="169efb02-ef71-4be0-be80-ca561af3d681" containerID="459edf88a0b29de4992f041f53aa6bf172fdf5e3b29bdac7b4195da4e0336c70" exitCode=0 Oct 30 22:24:21.465304 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.465216 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl" Oct 30 22:24:21.465304 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.465261 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl" event={"ID":"169efb02-ef71-4be0-be80-ca561af3d681","Type":"ContainerDied","Data":"459edf88a0b29de4992f041f53aa6bf172fdf5e3b29bdac7b4195da4e0336c70"} Oct 30 22:24:21.465392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.465333 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl" event={"ID":"169efb02-ef71-4be0-be80-ca561af3d681","Type":"ContainerDied","Data":"96568b08496ea4eb8d470b63629f47d71a3745e8a34bdc93ee6740954ed30549"} Oct 30 22:24:21.475661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.475641 4335 scope.go:122] "RemoveContainer" containerID="5e251bf9381cf7eec7b5f05e2a570468c2673393623560121ae83c1d63c54483" Oct 30 22:24:21.475955 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:24:21.475936 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e251bf9381cf7eec7b5f05e2a570468c2673393623560121ae83c1d63c54483\": container with ID starting with 5e251bf9381cf7eec7b5f05e2a570468c2673393623560121ae83c1d63c54483 not found: ID does not exist" containerID="5e251bf9381cf7eec7b5f05e2a570468c2673393623560121ae83c1d63c54483" Oct 30 22:24:21.476041 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.475966 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e251bf9381cf7eec7b5f05e2a570468c2673393623560121ae83c1d63c54483"} err="failed to get container status \"5e251bf9381cf7eec7b5f05e2a570468c2673393623560121ae83c1d63c54483\": rpc error: code = NotFound desc = could not find container \"5e251bf9381cf7eec7b5f05e2a570468c2673393623560121ae83c1d63c54483\": container with ID starting with 5e251bf9381cf7eec7b5f05e2a570468c2673393623560121ae83c1d63c54483 not found: ID does not exist" Oct 30 22:24:21.476041 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.475990 4335 scope.go:122] "RemoveContainer" containerID="459edf88a0b29de4992f041f53aa6bf172fdf5e3b29bdac7b4195da4e0336c70" Oct 30 22:24:21.489485 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.489463 4335 scope.go:122] "RemoveContainer" containerID="459edf88a0b29de4992f041f53aa6bf172fdf5e3b29bdac7b4195da4e0336c70" Oct 30 22:24:21.489983 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:24:21.489962 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"459edf88a0b29de4992f041f53aa6bf172fdf5e3b29bdac7b4195da4e0336c70\": container with ID starting with 459edf88a0b29de4992f041f53aa6bf172fdf5e3b29bdac7b4195da4e0336c70 not found: ID does not exist" containerID="459edf88a0b29de4992f041f53aa6bf172fdf5e3b29bdac7b4195da4e0336c70" Oct 30 22:24:21.490080 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.489991 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"459edf88a0b29de4992f041f53aa6bf172fdf5e3b29bdac7b4195da4e0336c70"} err="failed to get container status \"459edf88a0b29de4992f041f53aa6bf172fdf5e3b29bdac7b4195da4e0336c70\": rpc error: code = NotFound desc = could not find container \"459edf88a0b29de4992f041f53aa6bf172fdf5e3b29bdac7b4195da4e0336c70\": container with ID starting with 459edf88a0b29de4992f041f53aa6bf172fdf5e3b29bdac7b4195da4e0336c70 not found: ID does not exist" Oct 30 22:24:21.591996 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.591969 4335 patch_prober.go:29] interesting pod/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 30 22:24:21.592172 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.592032 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" podUID="58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 30 22:24:21.620855 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.620829 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9" Oct 30 22:24:21.634476 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.634441 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7"] Oct 30 22:24:21.639230 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.639198 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e3324355-7d8d998887-k9ht7"] Oct 30 22:24:21.753766 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.753687 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl"] Oct 30 22:24:21.768673 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:21.763571 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e3324355-5b75b48fcf-42xgl"] Oct 30 22:24:22.030466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:22.030431 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9"] Oct 30 22:24:22.477258 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:22.477227 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9" event={"ID":"4dd15165-4075-470b-92ab-8246e2fc26b4","Type":"ContainerStarted","Data":"ce7bbd327232384b9057c67457918a4478ff6f612ba40ef7ca7fbdbeeee2ffcd"} Oct 30 22:24:22.591626 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:22.591576 4335 patch_prober.go:29] interesting pod/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 30 22:24:22.591786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:22.591694 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" podUID="58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 30 22:24:23.062869 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:23.062837 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="169efb02-ef71-4be0-be80-ca561af3d681" path="/var/lib/kubelet/pods/169efb02-ef71-4be0-be80-ca561af3d681/volumes" Oct 30 22:24:23.063362 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:23.063346 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="770e0179-e20c-40ab-83f7-b8367aa0afb1" path="/var/lib/kubelet/pods/770e0179-e20c-40ab-83f7-b8367aa0afb1/volumes" Oct 30 22:24:23.490285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:23.490196 4335 generic.go:356] "Generic (PLEG): container finished" podID="4dd15165-4075-470b-92ab-8246e2fc26b4" containerID="e916455cc89f9a74ff46b2936652c2c771ed33180026f6e6fac3b03ded278e79" exitCode=0 Oct 30 22:24:23.490443 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:23.490279 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9" event={"ID":"4dd15165-4075-470b-92ab-8246e2fc26b4","Type":"ContainerDied","Data":"e916455cc89f9a74ff46b2936652c2c771ed33180026f6e6fac3b03ded278e79"} Oct 30 22:24:23.591489 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:23.591449 4335 patch_prober.go:29] interesting pod/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 30 22:24:23.591716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:23.591498 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" podUID="58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 30 22:24:24.596890 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:24.596864 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:24:24.598107 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:24.598085 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:24:24.686054 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:24.686032 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9" Oct 30 22:24:24.817755 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:24.817723 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd15165-4075-470b-92ab-8246e2fc26b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd15165-4075-470b-92ab-8246e2fc26b4-maas-api-xks-ca\") pod \"4dd15165-4075-470b-92ab-8246e2fc26b4\" (UID: \"4dd15165-4075-470b-92ab-8246e2fc26b4\") " Oct 30 22:24:24.817962 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:24.817797 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd15165-4075-470b-92ab-8246e2fc26b4-kube-api-access-gdftj\" (UniqueName: \"kubernetes.io/projected/4dd15165-4075-470b-92ab-8246e2fc26b4-kube-api-access-gdftj\") pod \"4dd15165-4075-470b-92ab-8246e2fc26b4\" (UID: \"4dd15165-4075-470b-92ab-8246e2fc26b4\") " Oct 30 22:24:24.818038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:24.817977 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd15165-4075-470b-92ab-8246e2fc26b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd15165-4075-470b-92ab-8246e2fc26b4-maas-api-openshift-ca\") pod \"4dd15165-4075-470b-92ab-8246e2fc26b4\" (UID: \"4dd15165-4075-470b-92ab-8246e2fc26b4\") " Oct 30 22:24:24.818587 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:24.818559 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd15165-4075-470b-92ab-8246e2fc26b4-maas-api-openshift-ca" pod "4dd15165-4075-470b-92ab-8246e2fc26b4" (UID: "4dd15165-4075-470b-92ab-8246e2fc26b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:24.823632 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:24.820506 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd15165-4075-470b-92ab-8246e2fc26b4-maas-api-xks-ca" pod "4dd15165-4075-470b-92ab-8246e2fc26b4" (UID: "4dd15165-4075-470b-92ab-8246e2fc26b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:24.823632 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:24.821375 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd15165-4075-470b-92ab-8246e2fc26b4-kube-api-access-gdftj" pod "4dd15165-4075-470b-92ab-8246e2fc26b4" (UID: "4dd15165-4075-470b-92ab-8246e2fc26b4"). InnerVolumeSpecName "kube-api-access-gdftj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:24.927639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:24.919905 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdftj\" (UniqueName: \"kubernetes.io/projected/4dd15165-4075-470b-92ab-8246e2fc26b4-kube-api-access-gdftj\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:24.927639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:24.919947 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd15165-4075-470b-92ab-8246e2fc26b4-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:24.927639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:24.919963 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd15165-4075-470b-92ab-8246e2fc26b4-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:25.322143 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:24:25.322102 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:24:25.511786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.511698 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9" Oct 30 22:24:25.511786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.511734 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9" event={"ID":"4dd15165-4075-470b-92ab-8246e2fc26b4","Type":"ContainerDied","Data":"ce7bbd327232384b9057c67457918a4478ff6f612ba40ef7ca7fbdbeeee2ffcd"} Oct 30 22:24:25.511786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.511786 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce7bbd327232384b9057c67457918a4478ff6f612ba40ef7ca7fbdbeeee2ffcd" Oct 30 22:24:25.639864 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.639832 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg"] Oct 30 22:24:25.640893 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.640870 4335 cpu_manager.go:414] "containerMap: removing container" podUID="4dd15165-4075-470b-92ab-8246e2fc26b4" containerName="revoke-keys" Oct 30 22:24:25.640893 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.640893 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd15165-4075-470b-92ab-8246e2fc26b4" containerName="revoke-keys" Oct 30 22:24:25.641103 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.641090 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd15165-4075-470b-92ab-8246e2fc26b4" containerName="revoke-keys" Oct 30 22:24:25.645532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.645509 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg" Oct 30 22:24:25.697760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.697733 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:24:25.697986 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.697964 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:24:25.698717 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.698694 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:24:25.698839 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.698801 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:24:25.701590 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.701569 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg"] Oct 30 22:24:25.731260 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.731230 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c97c3b-4a65-4885-9d2e-f67fd1336947-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg\" (UID: \"c3c97c3b-4a65-4885-9d2e-f67fd1336947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg" Oct 30 22:24:25.731421 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.731278 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c97c3b-4a65-4885-9d2e-f67fd1336947-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg\" (UID: \"c3c97c3b-4a65-4885-9d2e-f67fd1336947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg" Oct 30 22:24:25.731551 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.731532 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx6jf\" (UniqueName: \"kubernetes.io/projected/c3c97c3b-4a65-4885-9d2e-f67fd1336947-kube-api-access-jx6jf\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg\" (UID: \"c3c97c3b-4a65-4885-9d2e-f67fd1336947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg" Oct 30 22:24:25.833584 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.833554 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c97c3b-4a65-4885-9d2e-f67fd1336947-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg\" (UID: \"c3c97c3b-4a65-4885-9d2e-f67fd1336947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg" Oct 30 22:24:25.833795 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.833590 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c97c3b-4a65-4885-9d2e-f67fd1336947-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg\" (UID: \"c3c97c3b-4a65-4885-9d2e-f67fd1336947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg" Oct 30 22:24:25.833795 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.833786 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx6jf\" (UniqueName: \"kubernetes.io/projected/c3c97c3b-4a65-4885-9d2e-f67fd1336947-kube-api-access-jx6jf\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg\" (UID: \"c3c97c3b-4a65-4885-9d2e-f67fd1336947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg" Oct 30 22:24:25.834424 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.834400 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c97c3b-4a65-4885-9d2e-f67fd1336947-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg\" (UID: \"c3c97c3b-4a65-4885-9d2e-f67fd1336947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg" Oct 30 22:24:25.836138 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.836118 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c97c3b-4a65-4885-9d2e-f67fd1336947-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg\" (UID: \"c3c97c3b-4a65-4885-9d2e-f67fd1336947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg" Oct 30 22:24:25.894515 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.894483 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx6jf\" (UniqueName: \"kubernetes.io/projected/c3c97c3b-4a65-4885-9d2e-f67fd1336947-kube-api-access-jx6jf\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg\" (UID: \"c3c97c3b-4a65-4885-9d2e-f67fd1336947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg" Oct 30 22:24:25.960755 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:25.960733 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg" Oct 30 22:24:26.355837 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:24:26.355800 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c97c3b_4a65_4885_9d2e_f67fd1336947.slice/crio-d13decd2f2331eac5ab632496e2acae726809ec958aa743265798d044c98d744 WatchSource:0}: Error finding container d13decd2f2331eac5ab632496e2acae726809ec958aa743265798d044c98d744: Status 404 returned error can't find the container with id d13decd2f2331eac5ab632496e2acae726809ec958aa743265798d044c98d744 Oct 30 22:24:26.357012 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:26.356981 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg"] Oct 30 22:24:26.524288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:26.524253 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg" event={"ID":"c3c97c3b-4a65-4885-9d2e-f67fd1336947","Type":"ContainerStarted","Data":"d13decd2f2331eac5ab632496e2acae726809ec958aa743265798d044c98d744"} Oct 30 22:24:27.539080 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:27.538987 4335 generic.go:356] "Generic (PLEG): container finished" podID="c3c97c3b-4a65-4885-9d2e-f67fd1336947" containerID="56344adb7fded7a6a3c8f246b2571cc2b69aa8fad4b0d74f7deffe05e947e3a3" exitCode=0 Oct 30 22:24:27.539513 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:27.539122 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg" event={"ID":"c3c97c3b-4a65-4885-9d2e-f67fd1336947","Type":"ContainerDied","Data":"56344adb7fded7a6a3c8f246b2571cc2b69aa8fad4b0d74f7deffe05e947e3a3"} Oct 30 22:24:28.114783 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:28.114750 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n"] Oct 30 22:24:28.115169 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:28.114973 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" podUID="967921e5-a600-4870-bb43-721aaffbad2d" containerName="istio-proxy" containerID="cri-o://7eef06ccfa664791c5427bf9825be891366256539deae56eb0395ad90792255b" gracePeriod=30 Oct 30 22:24:28.362470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:28.362438 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:24:28.420018 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:28.419943 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:24:28.744335 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:28.744315 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg" Oct 30 22:24:28.882454 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:28.882410 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3c97c3b-4a65-4885-9d2e-f67fd1336947-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c97c3b-4a65-4885-9d2e-f67fd1336947-maas-api-xks-ca\") pod \"c3c97c3b-4a65-4885-9d2e-f67fd1336947\" (UID: \"c3c97c3b-4a65-4885-9d2e-f67fd1336947\") " Oct 30 22:24:28.882702 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:28.882566 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3c97c3b-4a65-4885-9d2e-f67fd1336947-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c97c3b-4a65-4885-9d2e-f67fd1336947-maas-api-openshift-ca\") pod \"c3c97c3b-4a65-4885-9d2e-f67fd1336947\" (UID: \"c3c97c3b-4a65-4885-9d2e-f67fd1336947\") " Oct 30 22:24:28.882817 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:28.882792 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c97c3b-4a65-4885-9d2e-f67fd1336947-kube-api-access-jx6jf\" (UniqueName: \"kubernetes.io/projected/c3c97c3b-4a65-4885-9d2e-f67fd1336947-kube-api-access-jx6jf\") pod \"c3c97c3b-4a65-4885-9d2e-f67fd1336947\" (UID: \"c3c97c3b-4a65-4885-9d2e-f67fd1336947\") " Oct 30 22:24:28.883335 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:28.883294 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c97c3b-4a65-4885-9d2e-f67fd1336947-maas-api-openshift-ca" pod "c3c97c3b-4a65-4885-9d2e-f67fd1336947" (UID: "c3c97c3b-4a65-4885-9d2e-f67fd1336947"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:28.885454 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:28.885425 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3c97c3b-4a65-4885-9d2e-f67fd1336947-maas-api-xks-ca" pod "c3c97c3b-4a65-4885-9d2e-f67fd1336947" (UID: "c3c97c3b-4a65-4885-9d2e-f67fd1336947"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:28.886070 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:28.886039 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c97c3b-4a65-4885-9d2e-f67fd1336947-kube-api-access-jx6jf" pod "c3c97c3b-4a65-4885-9d2e-f67fd1336947" (UID: "c3c97c3b-4a65-4885-9d2e-f67fd1336947"). InnerVolumeSpecName "kube-api-access-jx6jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:28.984422 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:28.984365 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx6jf\" (UniqueName: \"kubernetes.io/projected/c3c97c3b-4a65-4885-9d2e-f67fd1336947-kube-api-access-jx6jf\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:28.984422 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:28.984389 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c97c3b-4a65-4885-9d2e-f67fd1336947-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:28.984422 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:28.984400 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c97c3b-4a65-4885-9d2e-f67fd1336947-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.572166 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:29.572134 4335 scope.go:122] "RemoveContainer" containerID="b6776e8ea5053f4465247a881e8aa494745be2e51836188c0a9af5b7f84ca7ce" Oct 30 22:24:29.574175 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:29.574105 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg" Oct 30 22:24:29.574330 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:29.574104 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg" event={"ID":"c3c97c3b-4a65-4885-9d2e-f67fd1336947","Type":"ContainerDied","Data":"d13decd2f2331eac5ab632496e2acae726809ec958aa743265798d044c98d744"} Oct 30 22:24:29.574330 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:29.574256 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d13decd2f2331eac5ab632496e2acae726809ec958aa743265798d044c98d744" Oct 30 22:24:29.583798 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:29.583770 4335 scope.go:122] "RemoveContainer" containerID="a7acea7405f1a8cddec02b81afd86f638b49cf7954fd71fbfe7c113f063857e5" Oct 30 22:24:29.593941 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:29.593919 4335 scope.go:122] "RemoveContainer" containerID="62239c8f2f3bb8bc09eb40e08e5ed673e087a1ee23c701e789a90165971a48fe" Oct 30 22:24:31.222810 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:31.222772 4335 patch_prober.go:29] interesting pod/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:24:31.223214 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:31.222830 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" podUID="967921e5-a600-4870-bb43-721aaffbad2d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:24:33.459555 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.459529 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:24:33.618528 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.618483 4335 generic.go:356] "Generic (PLEG): container finished" podID="967921e5-a600-4870-bb43-721aaffbad2d" containerID="7eef06ccfa664791c5427bf9825be891366256539deae56eb0395ad90792255b" exitCode=0 Oct 30 22:24:33.618684 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.618533 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" Oct 30 22:24:33.618684 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.618558 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" event={"ID":"967921e5-a600-4870-bb43-721aaffbad2d","Type":"ContainerDied","Data":"7eef06ccfa664791c5427bf9825be891366256539deae56eb0395ad90792255b"} Oct 30 22:24:33.618684 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.618663 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n" event={"ID":"967921e5-a600-4870-bb43-721aaffbad2d","Type":"ContainerDied","Data":"55c24638ee7a5bc23ab8ccf692cb03f13943214df9c62650c84c389cc8e96fb0"} Oct 30 22:24:33.618802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.618699 4335 scope.go:122] "RemoveContainer" containerID="7eef06ccfa664791c5427bf9825be891366256539deae56eb0395ad90792255b" Oct 30 22:24:33.630298 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.630281 4335 scope.go:122] "RemoveContainer" containerID="73f3917ebf1a46d077acfad490c34bc55de6a39d16003fda80d3eaebdb913436" Oct 30 22:24:33.640941 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.640921 4335 scope.go:122] "RemoveContainer" containerID="7eef06ccfa664791c5427bf9825be891366256539deae56eb0395ad90792255b" Oct 30 22:24:33.641323 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:24:33.641264 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eef06ccfa664791c5427bf9825be891366256539deae56eb0395ad90792255b\": container with ID starting with 7eef06ccfa664791c5427bf9825be891366256539deae56eb0395ad90792255b not found: ID does not exist" containerID="7eef06ccfa664791c5427bf9825be891366256539deae56eb0395ad90792255b" Oct 30 22:24:33.641323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.641294 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eef06ccfa664791c5427bf9825be891366256539deae56eb0395ad90792255b"} err="failed to get container status \"7eef06ccfa664791c5427bf9825be891366256539deae56eb0395ad90792255b\": rpc error: code = NotFound desc = could not find container \"7eef06ccfa664791c5427bf9825be891366256539deae56eb0395ad90792255b\": container with ID starting with 7eef06ccfa664791c5427bf9825be891366256539deae56eb0395ad90792255b not found: ID does not exist" Oct 30 22:24:33.641323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.641316 4335 scope.go:122] "RemoveContainer" containerID="73f3917ebf1a46d077acfad490c34bc55de6a39d16003fda80d3eaebdb913436" Oct 30 22:24:33.641663 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:24:33.641602 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f3917ebf1a46d077acfad490c34bc55de6a39d16003fda80d3eaebdb913436\": container with ID starting with 73f3917ebf1a46d077acfad490c34bc55de6a39d16003fda80d3eaebdb913436 not found: ID does not exist" containerID="73f3917ebf1a46d077acfad490c34bc55de6a39d16003fda80d3eaebdb913436" Oct 30 22:24:33.641814 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.641669 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f3917ebf1a46d077acfad490c34bc55de6a39d16003fda80d3eaebdb913436"} err="failed to get container status \"73f3917ebf1a46d077acfad490c34bc55de6a39d16003fda80d3eaebdb913436\": rpc error: code = NotFound desc = could not find container \"73f3917ebf1a46d077acfad490c34bc55de6a39d16003fda80d3eaebdb913436\": container with ID starting with 73f3917ebf1a46d077acfad490c34bc55de6a39d16003fda80d3eaebdb913436 not found: ID does not exist" Oct 30 22:24:33.657029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.657003 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-istio-data\") pod \"967921e5-a600-4870-bb43-721aaffbad2d\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " Oct 30 22:24:33.657131 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.657093 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-credential-socket\") pod \"967921e5-a600-4870-bb43-721aaffbad2d\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " Oct 30 22:24:33.657243 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.657224 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-istio-data" pod "967921e5-a600-4870-bb43-721aaffbad2d" (UID: "967921e5-a600-4870-bb43-721aaffbad2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:33.657292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.657277 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967921e5-a600-4870-bb43-721aaffbad2d-istio-token\" (UniqueName: \"kubernetes.io/projected/967921e5-a600-4870-bb43-721aaffbad2d-istio-token\") pod \"967921e5-a600-4870-bb43-721aaffbad2d\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " Oct 30 22:24:33.657332 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.657320 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-workload-certs\") pod \"967921e5-a600-4870-bb43-721aaffbad2d\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " Oct 30 22:24:33.657392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.657347 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-credential-socket" pod "967921e5-a600-4870-bb43-721aaffbad2d" (UID: "967921e5-a600-4870-bb43-721aaffbad2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:33.657392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.657352 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967921e5-a600-4870-bb43-721aaffbad2d-kube-api-access-tvlgq\" (UniqueName: \"kubernetes.io/projected/967921e5-a600-4870-bb43-721aaffbad2d-kube-api-access-tvlgq\") pod \"967921e5-a600-4870-bb43-721aaffbad2d\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " Oct 30 22:24:33.657506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.657410 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/967921e5-a600-4870-bb43-721aaffbad2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967921e5-a600-4870-bb43-721aaffbad2d-istio-podinfo\") pod \"967921e5-a600-4870-bb43-721aaffbad2d\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " Oct 30 22:24:33.657506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.657451 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-istio-envoy\") pod \"967921e5-a600-4870-bb43-721aaffbad2d\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " Oct 30 22:24:33.657631 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.657576 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/967921e5-a600-4870-bb43-721aaffbad2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967921e5-a600-4870-bb43-721aaffbad2d-istiod-ca-cert\") pod \"967921e5-a600-4870-bb43-721aaffbad2d\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " Oct 30 22:24:33.657631 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.657587 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-workload-certs" pod "967921e5-a600-4870-bb43-721aaffbad2d" (UID: "967921e5-a600-4870-bb43-721aaffbad2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:33.657756 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.657648 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-workload-socket\") pod \"967921e5-a600-4870-bb43-721aaffbad2d\" (UID: \"967921e5-a600-4870-bb43-721aaffbad2d\") " Oct 30 22:24:33.657973 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.657897 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967921e5-a600-4870-bb43-721aaffbad2d-istiod-ca-cert" pod "967921e5-a600-4870-bb43-721aaffbad2d" (UID: "967921e5-a600-4870-bb43-721aaffbad2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:33.657973 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.657938 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-workload-socket" pod "967921e5-a600-4870-bb43-721aaffbad2d" (UID: "967921e5-a600-4870-bb43-721aaffbad2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:33.658307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.658213 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:33.658307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.658239 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967921e5-a600-4870-bb43-721aaffbad2d-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:33.658307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.658254 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:33.658307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.658270 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:33.658307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.658284 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:33.660009 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.659979 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/967921e5-a600-4870-bb43-721aaffbad2d-istio-podinfo" pod "967921e5-a600-4870-bb43-721aaffbad2d" (UID: "967921e5-a600-4870-bb43-721aaffbad2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:24:33.660107 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.660085 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967921e5-a600-4870-bb43-721aaffbad2d-kube-api-access-tvlgq" pod "967921e5-a600-4870-bb43-721aaffbad2d" (UID: "967921e5-a600-4870-bb43-721aaffbad2d"). InnerVolumeSpecName "kube-api-access-tvlgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:33.660278 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.660253 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-istio-envoy" pod "967921e5-a600-4870-bb43-721aaffbad2d" (UID: "967921e5-a600-4870-bb43-721aaffbad2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:33.660365 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.660295 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967921e5-a600-4870-bb43-721aaffbad2d-istio-token" pod "967921e5-a600-4870-bb43-721aaffbad2d" (UID: "967921e5-a600-4870-bb43-721aaffbad2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:33.759717 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.759692 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967921e5-a600-4870-bb43-721aaffbad2d-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:33.759717 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.759718 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvlgq\" (UniqueName: \"kubernetes.io/projected/967921e5-a600-4870-bb43-721aaffbad2d-kube-api-access-tvlgq\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:33.759885 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.759734 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967921e5-a600-4870-bb43-721aaffbad2d-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:33.759885 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:33.759749 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967921e5-a600-4870-bb43-721aaffbad2d-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:34.045849 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:34.045818 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n"] Oct 30 22:24:34.051665 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:34.051641 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bdc5fe-openshift-default-766f7889cf-tq87n"] Oct 30 22:24:35.059856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:35.059823 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="967921e5-a600-4870-bb43-721aaffbad2d" path="/var/lib/kubelet/pods/967921e5-a600-4870-bb43-721aaffbad2d/volumes" Oct 30 22:24:35.064969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:35.064943 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r"] Oct 30 22:24:35.114744 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:35.114714 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3324355-8n99r"] Oct 30 22:24:37.059518 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:37.059487 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11603f83-ff6f-421c-92a7-dd29e0721270" path="/var/lib/kubelet/pods/11603f83-ff6f-421c-92a7-dd29e0721270/volumes" Oct 30 22:24:40.440225 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.440190 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn"] Oct 30 22:24:40.441092 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.441063 4335 cpu_manager.go:414] "containerMap: removing container" podUID="967921e5-a600-4870-bb43-721aaffbad2d" containerName="istio-proxy" Oct 30 22:24:40.441092 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.441089 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967921e5-a600-4870-bb43-721aaffbad2d" containerName="istio-proxy" Oct 30 22:24:40.441266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.441170 4335 cpu_manager.go:414] "containerMap: removing container" podUID="967921e5-a600-4870-bb43-721aaffbad2d" containerName="istio-proxy" Oct 30 22:24:40.441266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.441179 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967921e5-a600-4870-bb43-721aaffbad2d" containerName="istio-proxy" Oct 30 22:24:40.441266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.441199 4335 cpu_manager.go:414] "containerMap: removing container" podUID="c3c97c3b-4a65-4885-9d2e-f67fd1336947" containerName="revoke-keys" Oct 30 22:24:40.441266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.441207 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c97c3b-4a65-4885-9d2e-f67fd1336947" containerName="revoke-keys" Oct 30 22:24:40.441386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.441347 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="967921e5-a600-4870-bb43-721aaffbad2d" containerName="istio-proxy" Oct 30 22:24:40.441386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.441360 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c97c3b-4a65-4885-9d2e-f67fd1336947" containerName="revoke-keys" Oct 30 22:24:40.446458 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.446435 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn" Oct 30 22:24:40.502098 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.502070 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn"] Oct 30 22:24:40.546127 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.546043 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:24:40.546433 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.546414 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:24:40.546584 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.546449 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:24:40.546584 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.546475 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:24:40.548049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.548026 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc37148-c569-4d8d-92ff-6a5bd1776aed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn\" (UID: \"acc37148-c569-4d8d-92ff-6a5bd1776aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn" Oct 30 22:24:40.548132 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.548068 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86gz2\" (UniqueName: \"kubernetes.io/projected/acc37148-c569-4d8d-92ff-6a5bd1776aed-kube-api-access-86gz2\") pod \"maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn\" (UID: \"acc37148-c569-4d8d-92ff-6a5bd1776aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn" Oct 30 22:24:40.548216 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.548196 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc37148-c569-4d8d-92ff-6a5bd1776aed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn\" (UID: \"acc37148-c569-4d8d-92ff-6a5bd1776aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn" Oct 30 22:24:40.649143 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.649115 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc37148-c569-4d8d-92ff-6a5bd1776aed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn\" (UID: \"acc37148-c569-4d8d-92ff-6a5bd1776aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn" Oct 30 22:24:40.649269 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.649167 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86gz2\" (UniqueName: \"kubernetes.io/projected/acc37148-c569-4d8d-92ff-6a5bd1776aed-kube-api-access-86gz2\") pod \"maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn\" (UID: \"acc37148-c569-4d8d-92ff-6a5bd1776aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn" Oct 30 22:24:40.649269 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.649256 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc37148-c569-4d8d-92ff-6a5bd1776aed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn\" (UID: \"acc37148-c569-4d8d-92ff-6a5bd1776aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn" Oct 30 22:24:40.649871 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.649842 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc37148-c569-4d8d-92ff-6a5bd1776aed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn\" (UID: \"acc37148-c569-4d8d-92ff-6a5bd1776aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn" Oct 30 22:24:40.651915 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.651884 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc37148-c569-4d8d-92ff-6a5bd1776aed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn\" (UID: \"acc37148-c569-4d8d-92ff-6a5bd1776aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn" Oct 30 22:24:40.709630 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.709548 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86gz2\" (UniqueName: \"kubernetes.io/projected/acc37148-c569-4d8d-92ff-6a5bd1776aed-kube-api-access-86gz2\") pod \"maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn\" (UID: \"acc37148-c569-4d8d-92ff-6a5bd1776aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn" Oct 30 22:24:40.762570 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:40.762547 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn" Oct 30 22:24:41.155814 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:24:41.155785 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc37148_c569_4d8d_92ff_6a5bd1776aed.slice/crio-fc603c53643c3fdb6cd5a952c4dfe33170f431c121a46a3d9205114cee6762d3 WatchSource:0}: Error finding container fc603c53643c3fdb6cd5a952c4dfe33170f431c121a46a3d9205114cee6762d3: Status 404 returned error can't find the container with id fc603c53643c3fdb6cd5a952c4dfe33170f431c121a46a3d9205114cee6762d3 Oct 30 22:24:41.157469 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:41.157445 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn"] Oct 30 22:24:41.693912 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:41.693866 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn" event={"ID":"acc37148-c569-4d8d-92ff-6a5bd1776aed","Type":"ContainerStarted","Data":"e6696dcc4a73dd462008a75969a10dbcd6b8fbf8d4d179cf191a7cd37dff035a"} Oct 30 22:24:41.694315 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:41.693930 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn" event={"ID":"acc37148-c569-4d8d-92ff-6a5bd1776aed","Type":"ContainerStarted","Data":"fc603c53643c3fdb6cd5a952c4dfe33170f431c121a46a3d9205114cee6762d3"} Oct 30 22:24:41.829593 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:41.829517 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn" podStartSLOduration=1.829497572 podStartE2EDuration="1.829497572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:41.819603556 +0000 UTC m=+2414.920035291" watchObservedRunningTime="2026-10-30 22:24:41.829497572 +0000 UTC m=+2414.929929306" Oct 30 22:24:41.876503 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:41.876469 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95"] Oct 30 22:24:41.876772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:41.876740 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" podUID="4e974fd0-2c92-4626-9673-d202cc596fea" containerName="istio-proxy" containerID="cri-o://48acb51650dc8ee47ddc693c72a531b1e0656949e2ee57f0b530084ff069dd55" gracePeriod=30 Oct 30 22:24:42.708205 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:42.708170 4335 generic.go:356] "Generic (PLEG): container finished" podID="acc37148-c569-4d8d-92ff-6a5bd1776aed" containerID="e6696dcc4a73dd462008a75969a10dbcd6b8fbf8d4d179cf191a7cd37dff035a" exitCode=0 Oct 30 22:24:42.708665 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:42.708248 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn" event={"ID":"acc37148-c569-4d8d-92ff-6a5bd1776aed","Type":"ContainerDied","Data":"e6696dcc4a73dd462008a75969a10dbcd6b8fbf8d4d179cf191a7cd37dff035a"} Oct 30 22:24:43.880052 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:43.880027 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn" Oct 30 22:24:43.996126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:43.996063 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acc37148-c569-4d8d-92ff-6a5bd1776aed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc37148-c569-4d8d-92ff-6a5bd1776aed-maas-api-openshift-ca\") pod \"acc37148-c569-4d8d-92ff-6a5bd1776aed\" (UID: \"acc37148-c569-4d8d-92ff-6a5bd1776aed\") " Oct 30 22:24:43.996404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:43.996386 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acc37148-c569-4d8d-92ff-6a5bd1776aed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc37148-c569-4d8d-92ff-6a5bd1776aed-maas-api-xks-ca\") pod \"acc37148-c569-4d8d-92ff-6a5bd1776aed\" (UID: \"acc37148-c569-4d8d-92ff-6a5bd1776aed\") " Oct 30 22:24:43.996808 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:43.996723 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc37148-c569-4d8d-92ff-6a5bd1776aed-kube-api-access-86gz2\" (UniqueName: \"kubernetes.io/projected/acc37148-c569-4d8d-92ff-6a5bd1776aed-kube-api-access-86gz2\") pod \"acc37148-c569-4d8d-92ff-6a5bd1776aed\" (UID: \"acc37148-c569-4d8d-92ff-6a5bd1776aed\") " Oct 30 22:24:43.997713 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:43.996547 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc37148-c569-4d8d-92ff-6a5bd1776aed-maas-api-openshift-ca" pod "acc37148-c569-4d8d-92ff-6a5bd1776aed" (UID: "acc37148-c569-4d8d-92ff-6a5bd1776aed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:43.999440 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:43.999406 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc37148-c569-4d8d-92ff-6a5bd1776aed-maas-api-xks-ca" pod "acc37148-c569-4d8d-92ff-6a5bd1776aed" (UID: "acc37148-c569-4d8d-92ff-6a5bd1776aed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:44.003150 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:44.003056 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc37148-c569-4d8d-92ff-6a5bd1776aed-kube-api-access-86gz2" pod "acc37148-c569-4d8d-92ff-6a5bd1776aed" (UID: "acc37148-c569-4d8d-92ff-6a5bd1776aed"). InnerVolumeSpecName "kube-api-access-86gz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:44.098501 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:44.098436 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc37148-c569-4d8d-92ff-6a5bd1776aed-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:44.098501 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:44.098463 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc37148-c569-4d8d-92ff-6a5bd1776aed-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:44.098501 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:44.098478 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86gz2\" (UniqueName: \"kubernetes.io/projected/acc37148-c569-4d8d-92ff-6a5bd1776aed-kube-api-access-86gz2\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:44.728872 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:44.728840 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn" Oct 30 22:24:44.729051 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:44.728868 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn" event={"ID":"acc37148-c569-4d8d-92ff-6a5bd1776aed","Type":"ContainerDied","Data":"fc603c53643c3fdb6cd5a952c4dfe33170f431c121a46a3d9205114cee6762d3"} Oct 30 22:24:44.729051 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:44.728914 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc603c53643c3fdb6cd5a952c4dfe33170f431c121a46a3d9205114cee6762d3" Oct 30 22:24:44.916762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:44.916731 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn"] Oct 30 22:24:44.942737 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:44.942702 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b"] Oct 30 22:24:47.160694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.160653 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227"] Oct 30 22:24:47.183825 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.183803 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:24:47.211229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.211204 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2d8852-p6227"] Oct 30 22:24:47.240058 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.240033 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e974fd0-2c92-4626-9673-d202cc596fea-kube-api-access-pffm5\" (UniqueName: \"kubernetes.io/projected/4e974fd0-2c92-4626-9673-d202cc596fea-kube-api-access-pffm5\") pod \"4e974fd0-2c92-4626-9673-d202cc596fea\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " Oct 30 22:24:47.240157 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.240092 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-workload-certs\") pod \"4e974fd0-2c92-4626-9673-d202cc596fea\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " Oct 30 22:24:47.240229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.240212 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e974fd0-2c92-4626-9673-d202cc596fea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e974fd0-2c92-4626-9673-d202cc596fea-istiod-ca-cert\") pod \"4e974fd0-2c92-4626-9673-d202cc596fea\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " Oct 30 22:24:47.240405 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.240384 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-workload-certs" pod "4e974fd0-2c92-4626-9673-d202cc596fea" (UID: "4e974fd0-2c92-4626-9673-d202cc596fea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:47.240838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.240575 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e974fd0-2c92-4626-9673-d202cc596fea-istiod-ca-cert" pod "4e974fd0-2c92-4626-9673-d202cc596fea" (UID: "4e974fd0-2c92-4626-9673-d202cc596fea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:47.241020 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.240386 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e974fd0-2c92-4626-9673-d202cc596fea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e974fd0-2c92-4626-9673-d202cc596fea-istio-podinfo\") pod \"4e974fd0-2c92-4626-9673-d202cc596fea\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " Oct 30 22:24:47.241145 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.241059 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-workload-socket\") pod \"4e974fd0-2c92-4626-9673-d202cc596fea\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " Oct 30 22:24:47.241211 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.241147 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-credential-socket\") pod \"4e974fd0-2c92-4626-9673-d202cc596fea\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " Oct 30 22:24:47.241211 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.241183 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-istio-envoy\") pod \"4e974fd0-2c92-4626-9673-d202cc596fea\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " Oct 30 22:24:47.241327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.241208 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e974fd0-2c92-4626-9673-d202cc596fea-istio-token\" (UniqueName: \"kubernetes.io/projected/4e974fd0-2c92-4626-9673-d202cc596fea-istio-token\") pod \"4e974fd0-2c92-4626-9673-d202cc596fea\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " Oct 30 22:24:47.241327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.241237 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-istio-data\") pod \"4e974fd0-2c92-4626-9673-d202cc596fea\" (UID: \"4e974fd0-2c92-4626-9673-d202cc596fea\") " Oct 30 22:24:47.241436 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.241363 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-workload-socket" pod "4e974fd0-2c92-4626-9673-d202cc596fea" (UID: "4e974fd0-2c92-4626-9673-d202cc596fea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:47.241862 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.241844 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e974fd0-2c92-4626-9673-d202cc596fea-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:47.241997 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.241984 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:47.242085 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.242059 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-istio-data" pod "4e974fd0-2c92-4626-9673-d202cc596fea" (UID: "4e974fd0-2c92-4626-9673-d202cc596fea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:47.242146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.242090 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-credential-socket" pod "4e974fd0-2c92-4626-9673-d202cc596fea" (UID: "4e974fd0-2c92-4626-9673-d202cc596fea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:47.242215 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.242075 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:47.242903 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.242883 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e974fd0-2c92-4626-9673-d202cc596fea-istio-podinfo" pod "4e974fd0-2c92-4626-9673-d202cc596fea" (UID: "4e974fd0-2c92-4626-9673-d202cc596fea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:24:47.243052 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.243029 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e974fd0-2c92-4626-9673-d202cc596fea-kube-api-access-pffm5" pod "4e974fd0-2c92-4626-9673-d202cc596fea" (UID: "4e974fd0-2c92-4626-9673-d202cc596fea"). InnerVolumeSpecName "kube-api-access-pffm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:47.243583 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.243562 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-istio-envoy" pod "4e974fd0-2c92-4626-9673-d202cc596fea" (UID: "4e974fd0-2c92-4626-9673-d202cc596fea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:47.243914 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.243898 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e974fd0-2c92-4626-9673-d202cc596fea-istio-token" pod "4e974fd0-2c92-4626-9673-d202cc596fea" (UID: "4e974fd0-2c92-4626-9673-d202cc596fea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:47.343754 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.343701 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e974fd0-2c92-4626-9673-d202cc596fea-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:47.343754 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.343749 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:47.343754 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.343764 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:47.344038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.343778 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e974fd0-2c92-4626-9673-d202cc596fea-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:47.344038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.343791 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e974fd0-2c92-4626-9673-d202cc596fea-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:47.344038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.343804 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pffm5\" (UniqueName: \"kubernetes.io/projected/4e974fd0-2c92-4626-9673-d202cc596fea-kube-api-access-pffm5\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:47.769651 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.769586 4335 generic.go:356] "Generic (PLEG): container finished" podID="4e974fd0-2c92-4626-9673-d202cc596fea" containerID="48acb51650dc8ee47ddc693c72a531b1e0656949e2ee57f0b530084ff069dd55" exitCode=0 Oct 30 22:24:47.769831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.769663 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" Oct 30 22:24:47.769831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.769684 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" event={"ID":"4e974fd0-2c92-4626-9673-d202cc596fea","Type":"ContainerDied","Data":"48acb51650dc8ee47ddc693c72a531b1e0656949e2ee57f0b530084ff069dd55"} Oct 30 22:24:47.769831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.769743 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95" event={"ID":"4e974fd0-2c92-4626-9673-d202cc596fea","Type":"ContainerDied","Data":"8950b8e3d18c71e47f002ee5b72b601ab78666718d2d7e6b0b18bdeeacd18859"} Oct 30 22:24:47.769831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.769774 4335 scope.go:122] "RemoveContainer" containerID="48acb51650dc8ee47ddc693c72a531b1e0656949e2ee57f0b530084ff069dd55" Oct 30 22:24:47.782090 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.782073 4335 scope.go:122] "RemoveContainer" containerID="d9da18cde1e801e291e005dd56c19ad7a9520de55b4d855408e7a44e5f847367" Oct 30 22:24:47.792380 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.792356 4335 scope.go:122] "RemoveContainer" containerID="48acb51650dc8ee47ddc693c72a531b1e0656949e2ee57f0b530084ff069dd55" Oct 30 22:24:47.792656 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:24:47.792636 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48acb51650dc8ee47ddc693c72a531b1e0656949e2ee57f0b530084ff069dd55\": container with ID starting with 48acb51650dc8ee47ddc693c72a531b1e0656949e2ee57f0b530084ff069dd55 not found: ID does not exist" containerID="48acb51650dc8ee47ddc693c72a531b1e0656949e2ee57f0b530084ff069dd55" Oct 30 22:24:47.792738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.792663 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48acb51650dc8ee47ddc693c72a531b1e0656949e2ee57f0b530084ff069dd55"} err="failed to get container status \"48acb51650dc8ee47ddc693c72a531b1e0656949e2ee57f0b530084ff069dd55\": rpc error: code = NotFound desc = could not find container \"48acb51650dc8ee47ddc693c72a531b1e0656949e2ee57f0b530084ff069dd55\": container with ID starting with 48acb51650dc8ee47ddc693c72a531b1e0656949e2ee57f0b530084ff069dd55 not found: ID does not exist" Oct 30 22:24:47.792738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.792689 4335 scope.go:122] "RemoveContainer" containerID="d9da18cde1e801e291e005dd56c19ad7a9520de55b4d855408e7a44e5f847367" Oct 30 22:24:47.792922 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:24:47.792907 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9da18cde1e801e291e005dd56c19ad7a9520de55b4d855408e7a44e5f847367\": container with ID starting with d9da18cde1e801e291e005dd56c19ad7a9520de55b4d855408e7a44e5f847367 not found: ID does not exist" containerID="d9da18cde1e801e291e005dd56c19ad7a9520de55b4d855408e7a44e5f847367" Oct 30 22:24:47.792966 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.792926 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9da18cde1e801e291e005dd56c19ad7a9520de55b4d855408e7a44e5f847367"} err="failed to get container status \"d9da18cde1e801e291e005dd56c19ad7a9520de55b4d855408e7a44e5f847367\": rpc error: code = NotFound desc = could not find container \"d9da18cde1e801e291e005dd56c19ad7a9520de55b4d855408e7a44e5f847367\": container with ID starting with d9da18cde1e801e291e005dd56c19ad7a9520de55b4d855408e7a44e5f847367 not found: ID does not exist" Oct 30 22:24:47.940043 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.940000 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95"] Oct 30 22:24:47.946102 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:47.946066 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e3324355-openshift-default-58dcd88df7-gwd95"] Oct 30 22:24:49.059131 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:49.059094 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e974fd0-2c92-4626-9673-d202cc596fea" path="/var/lib/kubelet/pods/4e974fd0-2c92-4626-9673-d202cc596fea/volumes" Oct 30 22:24:49.059658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:49.059533 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="585b415c-1e7a-4d41-886c-7c33110bfc31" path="/var/lib/kubelet/pods/585b415c-1e7a-4d41-886c-7c33110bfc31/volumes" Oct 30 22:24:51.481512 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.481479 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b"] Oct 30 22:24:51.640593 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.640562 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7"] Oct 30 22:24:51.641221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.641205 4335 cpu_manager.go:414] "containerMap: removing container" podUID="4e974fd0-2c92-4626-9673-d202cc596fea" containerName="istio-proxy" Oct 30 22:24:51.641221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.641221 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e974fd0-2c92-4626-9673-d202cc596fea" containerName="istio-proxy" Oct 30 22:24:51.641367 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.641240 4335 cpu_manager.go:414] "containerMap: removing container" podUID="4e974fd0-2c92-4626-9673-d202cc596fea" containerName="istio-proxy" Oct 30 22:24:51.641367 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.641245 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e974fd0-2c92-4626-9673-d202cc596fea" containerName="istio-proxy" Oct 30 22:24:51.641367 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.641253 4335 cpu_manager.go:414] "containerMap: removing container" podUID="acc37148-c569-4d8d-92ff-6a5bd1776aed" containerName="revoke-keys" Oct 30 22:24:51.641367 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.641259 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc37148-c569-4d8d-92ff-6a5bd1776aed" containerName="revoke-keys" Oct 30 22:24:51.641579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.641376 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e974fd0-2c92-4626-9673-d202cc596fea" containerName="istio-proxy" Oct 30 22:24:51.641579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.641386 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e974fd0-2c92-4626-9673-d202cc596fea" containerName="istio-proxy" Oct 30 22:24:51.641579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.641391 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc37148-c569-4d8d-92ff-6a5bd1776aed" containerName="revoke-keys" Oct 30 22:24:51.641579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.641403 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="967921e5-a600-4870-bb43-721aaffbad2d" containerName="istio-proxy" Oct 30 22:24:51.646547 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.646526 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7" Oct 30 22:24:51.694994 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.694967 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eb3c91f-8xsj7\" (UID: \"374b8c7b-ddb6-4209-b099-4cfa35b82a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7" Oct 30 22:24:51.695109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.695018 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k687h\" (UniqueName: \"kubernetes.io/projected/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-kube-api-access-k687h\") pod \"maas-api-revoke-sub-4eb3c91f-8xsj7\" (UID: \"374b8c7b-ddb6-4209-b099-4cfa35b82a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7" Oct 30 22:24:51.695210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.695189 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eb3c91f-8xsj7\" (UID: \"374b8c7b-ddb6-4209-b099-4cfa35b82a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7" Oct 30 22:24:51.701249 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.701183 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:24:51.701404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.701354 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:24:51.706377 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.706352 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw"] Oct 30 22:24:51.711681 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.711659 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7"] Oct 30 22:24:51.711793 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.711768 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw" Oct 30 22:24:51.737222 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.737167 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:24:51.737222 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.737211 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:24:51.771503 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.766729 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw"] Oct 30 22:24:51.797198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.796774 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eb3c91f-8xsj7\" (UID: \"374b8c7b-ddb6-4209-b099-4cfa35b82a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7" Oct 30 22:24:51.797198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.796929 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eb3c91f-8xsj7\" (UID: \"374b8c7b-ddb6-4209-b099-4cfa35b82a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7" Oct 30 22:24:51.797198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.796964 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqf2n\" (UniqueName: \"kubernetes.io/projected/1d41c004-bc10-4167-b242-cd963585abbe-kube-api-access-rqf2n\") pod \"maas-api-revoke-sub-d22538ad-8g2jw\" (UID: \"1d41c004-bc10-4167-b242-cd963585abbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw" Oct 30 22:24:51.797198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.797014 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k687h\" (UniqueName: \"kubernetes.io/projected/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-kube-api-access-k687h\") pod \"maas-api-revoke-sub-4eb3c91f-8xsj7\" (UID: \"374b8c7b-ddb6-4209-b099-4cfa35b82a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7" Oct 30 22:24:51.802878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.799347 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eb3c91f-8xsj7\" (UID: \"374b8c7b-ddb6-4209-b099-4cfa35b82a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7" Oct 30 22:24:51.802878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.799556 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d41c004-bc10-4167-b242-cd963585abbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d22538ad-8g2jw\" (UID: \"1d41c004-bc10-4167-b242-cd963585abbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw" Oct 30 22:24:51.802878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.799656 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d41c004-bc10-4167-b242-cd963585abbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d22538ad-8g2jw\" (UID: \"1d41c004-bc10-4167-b242-cd963585abbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw" Oct 30 22:24:51.822961 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.822349 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eb3c91f-8xsj7\" (UID: \"374b8c7b-ddb6-4209-b099-4cfa35b82a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7" Oct 30 22:24:51.859341 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.859315 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k687h\" (UniqueName: \"kubernetes.io/projected/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-kube-api-access-k687h\") pod \"maas-api-revoke-sub-4eb3c91f-8xsj7\" (UID: \"374b8c7b-ddb6-4209-b099-4cfa35b82a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7" Oct 30 22:24:51.908635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.906925 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d41c004-bc10-4167-b242-cd963585abbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d22538ad-8g2jw\" (UID: \"1d41c004-bc10-4167-b242-cd963585abbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw" Oct 30 22:24:51.908635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.908095 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d41c004-bc10-4167-b242-cd963585abbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d22538ad-8g2jw\" (UID: \"1d41c004-bc10-4167-b242-cd963585abbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw" Oct 30 22:24:51.912561 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.910072 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d41c004-bc10-4167-b242-cd963585abbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d22538ad-8g2jw\" (UID: \"1d41c004-bc10-4167-b242-cd963585abbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw" Oct 30 22:24:51.912561 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.910515 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqf2n\" (UniqueName: \"kubernetes.io/projected/1d41c004-bc10-4167-b242-cd963585abbe-kube-api-access-rqf2n\") pod \"maas-api-revoke-sub-d22538ad-8g2jw\" (UID: \"1d41c004-bc10-4167-b242-cd963585abbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw" Oct 30 22:24:51.942640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.935189 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d41c004-bc10-4167-b242-cd963585abbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d22538ad-8g2jw\" (UID: \"1d41c004-bc10-4167-b242-cd963585abbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw" Oct 30 22:24:51.961592 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.961559 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7" Oct 30 22:24:51.971406 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:51.971383 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqf2n\" (UniqueName: \"kubernetes.io/projected/1d41c004-bc10-4167-b242-cd963585abbe-kube-api-access-rqf2n\") pod \"maas-api-revoke-sub-d22538ad-8g2jw\" (UID: \"1d41c004-bc10-4167-b242-cd963585abbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw" Oct 30 22:24:52.038189 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:52.034547 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw" Oct 30 22:24:52.381493 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:52.381466 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7"] Oct 30 22:24:52.382670 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:24:52.382644 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374b8c7b_ddb6_4209_b099_4cfa35b82a2e.slice/crio-df5c592cf836f5b8a583c342b9569f1bd3ae83a11d80b56500f5e2e2fbe6d66e WatchSource:0}: Error finding container df5c592cf836f5b8a583c342b9569f1bd3ae83a11d80b56500f5e2e2fbe6d66e: Status 404 returned error can't find the container with id df5c592cf836f5b8a583c342b9569f1bd3ae83a11d80b56500f5e2e2fbe6d66e Oct 30 22:24:52.493757 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:52.493727 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw"] Oct 30 22:24:52.494077 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:24:52.493798 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d41c004_bc10_4167_b242_cd963585abbe.slice/crio-5192ef1602e25d21581a1de9df221d33791261a169cafa204666a626438d7e72 WatchSource:0}: Error finding container 5192ef1602e25d21581a1de9df221d33791261a169cafa204666a626438d7e72: Status 404 returned error can't find the container with id 5192ef1602e25d21581a1de9df221d33791261a169cafa204666a626438d7e72 Oct 30 22:24:52.844335 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:52.844303 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7" event={"ID":"374b8c7b-ddb6-4209-b099-4cfa35b82a2e","Type":"ContainerStarted","Data":"df5c592cf836f5b8a583c342b9569f1bd3ae83a11d80b56500f5e2e2fbe6d66e"} Oct 30 22:24:52.846015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:52.845992 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw" event={"ID":"1d41c004-bc10-4167-b242-cd963585abbe","Type":"ContainerStarted","Data":"e5c4bd93cc9311e2cfa3c2dd6509dda274c88e51831ff8e356a621d466a14fe2"} Oct 30 22:24:52.846118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:52.846032 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw" event={"ID":"1d41c004-bc10-4167-b242-cd963585abbe","Type":"ContainerStarted","Data":"5192ef1602e25d21581a1de9df221d33791261a169cafa204666a626438d7e72"} Oct 30 22:24:53.006745 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.000011 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw" podStartSLOduration=1.999987811 podStartE2EDuration="1.999987811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:52.989196831 +0000 UTC m=+2426.089628566" watchObservedRunningTime="2026-10-30 22:24:52.999987811 +0000 UTC m=+2426.100419546" Oct 30 22:24:53.274381 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.274311 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4"] Oct 30 22:24:53.278511 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.278490 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.343642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.328468 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-envoy\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.343946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.343918 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-podinfo\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.344154 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.344134 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-token\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.352640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.344300 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-workload-certs\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.352951 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.352924 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1657e3aa-0cfe-49e5-b8ce-21888a334461-istiod-ca-cert\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.418642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.407770 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-workload-socket\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.418642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.407844 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-credential-socket\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.418642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.407877 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-data\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.418642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.361203 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-026c11-openshift-default-dockercfg-wzj69\"" Oct 30 22:24:53.418642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.367255 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4"] Oct 30 22:24:53.418642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.408247 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcqjf\" (UniqueName: \"kubernetes.io/projected/1657e3aa-0cfe-49e5-b8ce-21888a334461-kube-api-access-fcqjf\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.509228 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.509200 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-workload-certs\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.509696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.509265 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1657e3aa-0cfe-49e5-b8ce-21888a334461-istiod-ca-cert\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.509696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.509301 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-workload-socket\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.509696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.509331 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-credential-socket\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.509696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.509357 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-data\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.509696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.509507 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqjf\" (UniqueName: \"kubernetes.io/projected/1657e3aa-0cfe-49e5-b8ce-21888a334461-kube-api-access-fcqjf\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.509696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.509575 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-envoy\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.509696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.509634 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-podinfo\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.509696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.509632 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-workload-certs\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.509696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.509672 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-token\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.510265 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.509894 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-data\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.510265 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.510026 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1657e3aa-0cfe-49e5-b8ce-21888a334461-istiod-ca-cert\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.510265 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.510131 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-workload-socket\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.510442 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.510300 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-credential-socket\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.512594 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.512574 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-envoy\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.512740 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.512656 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-podinfo\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.575879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.575851 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-token\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.576126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.576102 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcqjf\" (UniqueName: \"kubernetes.io/projected/1657e3aa-0cfe-49e5-b8ce-21888a334461-kube-api-access-fcqjf\") pod \"e2e-inf-a-026c11-openshift-default-5884675f97-6szq4\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.593152 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.592891 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:53.857318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.857281 4335 generic.go:356] "Generic (PLEG): container finished" podID="374b8c7b-ddb6-4209-b099-4cfa35b82a2e" containerID="42ee60300ddfd1954b7068feb443bb30cf5cc25220dea0111910611dbadd994d" exitCode=0 Oct 30 22:24:53.857453 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.857356 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7" event={"ID":"374b8c7b-ddb6-4209-b099-4cfa35b82a2e","Type":"ContainerDied","Data":"42ee60300ddfd1954b7068feb443bb30cf5cc25220dea0111910611dbadd994d"} Oct 30 22:24:53.859108 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.859087 4335 generic.go:356] "Generic (PLEG): container finished" podID="1d41c004-bc10-4167-b242-cd963585abbe" containerID="e5c4bd93cc9311e2cfa3c2dd6509dda274c88e51831ff8e356a621d466a14fe2" exitCode=0 Oct 30 22:24:53.859239 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:53.859168 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw" event={"ID":"1d41c004-bc10-4167-b242-cd963585abbe","Type":"ContainerDied","Data":"e5c4bd93cc9311e2cfa3c2dd6509dda274c88e51831ff8e356a621d466a14fe2"} Oct 30 22:24:54.042414 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:24:54.042385 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1657e3aa_0cfe_49e5_b8ce_21888a334461.slice/crio-c3c4b0265a0d2bae5c1fde466e04b239d87f1230a9634004ae6678468745408d WatchSource:0}: Error finding container c3c4b0265a0d2bae5c1fde466e04b239d87f1230a9634004ae6678468745408d: Status 404 returned error can't find the container with id c3c4b0265a0d2bae5c1fde466e04b239d87f1230a9634004ae6678468745408d Oct 30 22:24:54.043449 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:54.043421 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4"] Oct 30 22:24:54.051689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:54.051660 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:24:54.051802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:54.051733 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:24:54.051802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:54.051777 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:24:54.796797 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:54.796762 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n"] Oct 30 22:24:54.802051 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:54.802032 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n" Oct 30 22:24:54.829708 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:54.829669 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2kjk\" (UniqueName: \"kubernetes.io/projected/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-kube-api-access-j2kjk\") pod \"maas-api-revoke-keys-e2e-api-a-25237a-fgv6n\" (UID: \"5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n" Oct 30 22:24:54.829821 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:54.829796 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25237a-fgv6n\" (UID: \"5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n" Oct 30 22:24:54.830021 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:54.829999 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25237a-fgv6n\" (UID: \"5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n" Oct 30 22:24:54.856451 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:54.856424 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n"] Oct 30 22:24:54.871191 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:54.871157 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" event={"ID":"1657e3aa-0cfe-49e5-b8ce-21888a334461","Type":"ContainerStarted","Data":"f305f04e50ee079295e625005f01f05fef22490b4cc3c54a926f557a040df7b4"} Oct 30 22:24:54.871324 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:54.871214 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" event={"ID":"1657e3aa-0cfe-49e5-b8ce-21888a334461","Type":"ContainerStarted","Data":"c3c4b0265a0d2bae5c1fde466e04b239d87f1230a9634004ae6678468745408d"} Oct 30 22:24:54.931585 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:54.931545 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25237a-fgv6n\" (UID: \"5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n" Oct 30 22:24:54.931795 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:54.931724 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25237a-fgv6n\" (UID: \"5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n" Oct 30 22:24:54.931906 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:54.931828 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2kjk\" (UniqueName: \"kubernetes.io/projected/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-kube-api-access-j2kjk\") pod \"maas-api-revoke-keys-e2e-api-a-25237a-fgv6n\" (UID: \"5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n" Oct 30 22:24:54.933865 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:54.933807 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25237a-fgv6n\" (UID: \"5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n" Oct 30 22:24:54.935408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:54.935336 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25237a-fgv6n\" (UID: \"5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n" Oct 30 22:24:54.995393 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:54.995363 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2kjk\" (UniqueName: \"kubernetes.io/projected/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-kube-api-access-j2kjk\") pod \"maas-api-revoke-keys-e2e-api-a-25237a-fgv6n\" (UID: \"5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n" Oct 30 22:24:55.044546 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.044481 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" podStartSLOduration=2.044462745 podStartE2EDuration="2.044462745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:55.038510256 +0000 UTC m=+2428.138941988" watchObservedRunningTime="2026-10-30 22:24:55.044462745 +0000 UTC m=+2428.144894478" Oct 30 22:24:55.070943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.070920 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw" Oct 30 22:24:55.096585 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.096566 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7" Oct 30 22:24:55.116069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.116045 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n" Oct 30 22:24:55.134173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.134151 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-maas-api-xks-ca\") pod \"374b8c7b-ddb6-4209-b099-4cfa35b82a2e\" (UID: \"374b8c7b-ddb6-4209-b099-4cfa35b82a2e\") " Oct 30 22:24:55.134265 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.134235 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d41c004-bc10-4167-b242-cd963585abbe-kube-api-access-rqf2n\" (UniqueName: \"kubernetes.io/projected/1d41c004-bc10-4167-b242-cd963585abbe-kube-api-access-rqf2n\") pod \"1d41c004-bc10-4167-b242-cd963585abbe\" (UID: \"1d41c004-bc10-4167-b242-cd963585abbe\") " Oct 30 22:24:55.134377 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.134361 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-kube-api-access-k687h\" (UniqueName: \"kubernetes.io/projected/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-kube-api-access-k687h\") pod \"374b8c7b-ddb6-4209-b099-4cfa35b82a2e\" (UID: \"374b8c7b-ddb6-4209-b099-4cfa35b82a2e\") " Oct 30 22:24:55.134432 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.134417 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-maas-api-openshift-ca\") pod \"374b8c7b-ddb6-4209-b099-4cfa35b82a2e\" (UID: \"374b8c7b-ddb6-4209-b099-4cfa35b82a2e\") " Oct 30 22:24:55.134488 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.134475 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d41c004-bc10-4167-b242-cd963585abbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d41c004-bc10-4167-b242-cd963585abbe-maas-api-openshift-ca\") pod \"1d41c004-bc10-4167-b242-cd963585abbe\" (UID: \"1d41c004-bc10-4167-b242-cd963585abbe\") " Oct 30 22:24:55.134589 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.134567 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d41c004-bc10-4167-b242-cd963585abbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d41c004-bc10-4167-b242-cd963585abbe-maas-api-xks-ca\") pod \"1d41c004-bc10-4167-b242-cd963585abbe\" (UID: \"1d41c004-bc10-4167-b242-cd963585abbe\") " Oct 30 22:24:55.135343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.134965 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-maas-api-openshift-ca" pod "374b8c7b-ddb6-4209-b099-4cfa35b82a2e" (UID: "374b8c7b-ddb6-4209-b099-4cfa35b82a2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:55.135343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.135179 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:55.135829 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.135802 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d41c004-bc10-4167-b242-cd963585abbe-maas-api-openshift-ca" pod "1d41c004-bc10-4167-b242-cd963585abbe" (UID: "1d41c004-bc10-4167-b242-cd963585abbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:55.137122 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.137101 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-kube-api-access-k687h" pod "374b8c7b-ddb6-4209-b099-4cfa35b82a2e" (UID: "374b8c7b-ddb6-4209-b099-4cfa35b82a2e"). InnerVolumeSpecName "kube-api-access-k687h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:55.137204 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.137122 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-maas-api-xks-ca" pod "374b8c7b-ddb6-4209-b099-4cfa35b82a2e" (UID: "374b8c7b-ddb6-4209-b099-4cfa35b82a2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:55.137204 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.137160 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d41c004-bc10-4167-b242-cd963585abbe-kube-api-access-rqf2n" pod "1d41c004-bc10-4167-b242-cd963585abbe" (UID: "1d41c004-bc10-4167-b242-cd963585abbe"). InnerVolumeSpecName "kube-api-access-rqf2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:55.137317 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.137285 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d41c004-bc10-4167-b242-cd963585abbe-maas-api-xks-ca" pod "1d41c004-bc10-4167-b242-cd963585abbe" (UID: "1d41c004-bc10-4167-b242-cd963585abbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:55.237712 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.236706 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k687h\" (UniqueName: \"kubernetes.io/projected/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-kube-api-access-k687h\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:55.237712 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.236744 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d41c004-bc10-4167-b242-cd963585abbe-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:55.237712 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.236764 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d41c004-bc10-4167-b242-cd963585abbe-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:55.237712 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.236781 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/374b8c7b-ddb6-4209-b099-4cfa35b82a2e-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:55.237712 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.236794 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqf2n\" (UniqueName: \"kubernetes.io/projected/1d41c004-bc10-4167-b242-cd963585abbe-kube-api-access-rqf2n\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:55.334665 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:24:55.334572 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:24:55.594353 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.594280 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:55.600147 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.600124 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:55.616760 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:24:55.616728 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ddba4b4_abbc_472e_8b4f_2ea2aa876ed8.slice/crio-d8983ab9ea039565e552b69a90a2e9ceff94cf57322a7548431c3ae874d04194 WatchSource:0}: Error finding container d8983ab9ea039565e552b69a90a2e9ceff94cf57322a7548431c3ae874d04194: Status 404 returned error can't find the container with id d8983ab9ea039565e552b69a90a2e9ceff94cf57322a7548431c3ae874d04194 Oct 30 22:24:55.618143 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.618122 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n"] Oct 30 22:24:55.835499 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.835472 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c"] Oct 30 22:24:55.836150 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.836135 4335 cpu_manager.go:414] "containerMap: removing container" podUID="1d41c004-bc10-4167-b242-cd963585abbe" containerName="revoke-keys" Oct 30 22:24:55.836195 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.836151 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d41c004-bc10-4167-b242-cd963585abbe" containerName="revoke-keys" Oct 30 22:24:55.836195 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.836169 4335 cpu_manager.go:414] "containerMap: removing container" podUID="374b8c7b-ddb6-4209-b099-4cfa35b82a2e" containerName="revoke-keys" Oct 30 22:24:55.836195 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.836174 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="374b8c7b-ddb6-4209-b099-4cfa35b82a2e" containerName="revoke-keys" Oct 30 22:24:55.836297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.836266 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d41c004-bc10-4167-b242-cd963585abbe" containerName="revoke-keys" Oct 30 22:24:55.836297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.836277 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="374b8c7b-ddb6-4209-b099-4cfa35b82a2e" containerName="revoke-keys" Oct 30 22:24:55.841404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.841300 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c" Oct 30 22:24:55.882276 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.882252 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7" Oct 30 22:24:55.882439 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.882249 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7" event={"ID":"374b8c7b-ddb6-4209-b099-4cfa35b82a2e","Type":"ContainerDied","Data":"df5c592cf836f5b8a583c342b9569f1bd3ae83a11d80b56500f5e2e2fbe6d66e"} Oct 30 22:24:55.882507 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.882447 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df5c592cf836f5b8a583c342b9569f1bd3ae83a11d80b56500f5e2e2fbe6d66e" Oct 30 22:24:55.884039 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.884015 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw" Oct 30 22:24:55.884186 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.884065 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw" event={"ID":"1d41c004-bc10-4167-b242-cd963585abbe","Type":"ContainerDied","Data":"5192ef1602e25d21581a1de9df221d33791261a169cafa204666a626438d7e72"} Oct 30 22:24:55.884186 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.884134 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5192ef1602e25d21581a1de9df221d33791261a169cafa204666a626438d7e72" Oct 30 22:24:55.886402 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.886375 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n" event={"ID":"5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8","Type":"ContainerStarted","Data":"d8983ab9ea039565e552b69a90a2e9ceff94cf57322a7548431c3ae874d04194"} Oct 30 22:24:55.887124 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.886983 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:55.888693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.888655 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:24:55.904178 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.904156 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c"] Oct 30 22:24:55.946369 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.946098 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103a57d0-2aca-476e-805d-7c35c93eb04b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c\" (UID: \"103a57d0-2aca-476e-805d-7c35c93eb04b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c" Oct 30 22:24:55.946369 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.946308 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jkzl\" (UniqueName: \"kubernetes.io/projected/103a57d0-2aca-476e-805d-7c35c93eb04b-kube-api-access-9jkzl\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c\" (UID: \"103a57d0-2aca-476e-805d-7c35c93eb04b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c" Oct 30 22:24:55.946545 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:55.946434 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103a57d0-2aca-476e-805d-7c35c93eb04b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c\" (UID: \"103a57d0-2aca-476e-805d-7c35c93eb04b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c" Oct 30 22:24:56.047305 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:56.047279 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103a57d0-2aca-476e-805d-7c35c93eb04b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c\" (UID: \"103a57d0-2aca-476e-805d-7c35c93eb04b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c" Oct 30 22:24:56.047421 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:56.047338 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jkzl\" (UniqueName: \"kubernetes.io/projected/103a57d0-2aca-476e-805d-7c35c93eb04b-kube-api-access-9jkzl\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c\" (UID: \"103a57d0-2aca-476e-805d-7c35c93eb04b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c" Oct 30 22:24:56.047421 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:56.047391 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103a57d0-2aca-476e-805d-7c35c93eb04b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c\" (UID: \"103a57d0-2aca-476e-805d-7c35c93eb04b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c" Oct 30 22:24:56.047912 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:56.047886 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103a57d0-2aca-476e-805d-7c35c93eb04b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c\" (UID: \"103a57d0-2aca-476e-805d-7c35c93eb04b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c" Oct 30 22:24:56.049882 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:56.049859 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103a57d0-2aca-476e-805d-7c35c93eb04b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c\" (UID: \"103a57d0-2aca-476e-805d-7c35c93eb04b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c" Oct 30 22:24:56.108698 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:56.108592 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jkzl\" (UniqueName: \"kubernetes.io/projected/103a57d0-2aca-476e-805d-7c35c93eb04b-kube-api-access-9jkzl\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c\" (UID: \"103a57d0-2aca-476e-805d-7c35c93eb04b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c" Oct 30 22:24:56.158209 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:56.158186 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c" Oct 30 22:24:56.482671 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:56.482573 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" podUID="dd65776e-f29f-436a-aabb-e9e6b6d3497a" containerName="payload-processing" containerID="cri-o://3c8b283c688e0a53da0918edb87abe4a24c6323b112bc52e49f7e8a7496267b0" gracePeriod=25 Oct 30 22:24:56.551268 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:56.551240 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c"] Oct 30 22:24:56.558788 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:24:56.558764 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103a57d0_2aca_476e_805d_7c35c93eb04b.slice/crio-4de60ae6bdd2414699051a8eeb897ddabff0fd42ccafe1f643f4e40a062eb9d9 WatchSource:0}: Error finding container 4de60ae6bdd2414699051a8eeb897ddabff0fd42ccafe1f643f4e40a062eb9d9: Status 404 returned error can't find the container with id 4de60ae6bdd2414699051a8eeb897ddabff0fd42ccafe1f643f4e40a062eb9d9 Oct 30 22:24:56.896698 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:56.896660 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c" event={"ID":"103a57d0-2aca-476e-805d-7c35c93eb04b","Type":"ContainerStarted","Data":"4de60ae6bdd2414699051a8eeb897ddabff0fd42ccafe1f643f4e40a062eb9d9"} Oct 30 22:24:56.898674 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:56.898647 4335 generic.go:356] "Generic (PLEG): container finished" podID="5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8" containerID="6f640a772d55fad8f12ca784c6d78ac50fc7422c5dbeb0f5e2f33f3bd7d993fd" exitCode=0 Oct 30 22:24:56.898828 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:56.898710 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n" event={"ID":"5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8","Type":"ContainerDied","Data":"6f640a772d55fad8f12ca784c6d78ac50fc7422c5dbeb0f5e2f33f3bd7d993fd"} Oct 30 22:24:57.153636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:57.153523 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" podUID="dd65776e-f29f-436a-aabb-e9e6b6d3497a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:24:57.908959 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:57.908883 4335 generic.go:356] "Generic (PLEG): container finished" podID="103a57d0-2aca-476e-805d-7c35c93eb04b" containerID="2a3decaa595b0be204ab1359271dfc87fbd2862d6351edf3362bc1b8a44119e5" exitCode=0 Oct 30 22:24:57.908959 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:57.908952 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c" event={"ID":"103a57d0-2aca-476e-805d-7c35c93eb04b","Type":"ContainerDied","Data":"2a3decaa595b0be204ab1359271dfc87fbd2862d6351edf3362bc1b8a44119e5"} Oct 30 22:24:58.076116 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:58.076094 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n" Oct 30 22:24:58.174976 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:58.174916 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-kube-api-access-j2kjk\" (UniqueName: \"kubernetes.io/projected/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-kube-api-access-j2kjk\") pod \"5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8\" (UID: \"5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8\") " Oct 30 22:24:58.175111 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:58.174977 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-maas-api-xks-ca\") pod \"5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8\" (UID: \"5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8\") " Oct 30 22:24:58.175111 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:58.175022 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-maas-api-openshift-ca\") pod \"5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8\" (UID: \"5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8\") " Oct 30 22:24:58.175426 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:58.175396 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-maas-api-openshift-ca" pod "5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8" (UID: "5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:58.175662 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:58.175514 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:58.177183 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:58.177159 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-maas-api-xks-ca" pod "5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8" (UID: "5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:58.177298 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:58.177255 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-kube-api-access-j2kjk" pod "5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8" (UID: "5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8"). InnerVolumeSpecName "kube-api-access-j2kjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:58.276794 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:58.276763 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2kjk\" (UniqueName: \"kubernetes.io/projected/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-kube-api-access-j2kjk\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:58.276794 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:58.276792 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:58.923789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:58.922091 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n" event={"ID":"5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8","Type":"ContainerDied","Data":"d8983ab9ea039565e552b69a90a2e9ceff94cf57322a7548431c3ae874d04194"} Oct 30 22:24:58.923789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:58.922118 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n" Oct 30 22:24:58.923789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:58.922220 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8983ab9ea039565e552b69a90a2e9ceff94cf57322a7548431c3ae874d04194" Oct 30 22:24:59.128963 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:59.128942 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c" Oct 30 22:24:59.196074 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:59.196003 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103a57d0-2aca-476e-805d-7c35c93eb04b-kube-api-access-9jkzl\" (UniqueName: \"kubernetes.io/projected/103a57d0-2aca-476e-805d-7c35c93eb04b-kube-api-access-9jkzl\") pod \"103a57d0-2aca-476e-805d-7c35c93eb04b\" (UID: \"103a57d0-2aca-476e-805d-7c35c93eb04b\") " Oct 30 22:24:59.196215 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:59.196146 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103a57d0-2aca-476e-805d-7c35c93eb04b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103a57d0-2aca-476e-805d-7c35c93eb04b-maas-api-openshift-ca\") pod \"103a57d0-2aca-476e-805d-7c35c93eb04b\" (UID: \"103a57d0-2aca-476e-805d-7c35c93eb04b\") " Oct 30 22:24:59.196291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:59.196264 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/103a57d0-2aca-476e-805d-7c35c93eb04b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103a57d0-2aca-476e-805d-7c35c93eb04b-maas-api-xks-ca\") pod \"103a57d0-2aca-476e-805d-7c35c93eb04b\" (UID: \"103a57d0-2aca-476e-805d-7c35c93eb04b\") " Oct 30 22:24:59.196495 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:59.196474 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103a57d0-2aca-476e-805d-7c35c93eb04b-maas-api-openshift-ca" pod "103a57d0-2aca-476e-805d-7c35c93eb04b" (UID: "103a57d0-2aca-476e-805d-7c35c93eb04b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:59.196790 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:59.196770 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103a57d0-2aca-476e-805d-7c35c93eb04b-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:59.198414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:59.198385 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103a57d0-2aca-476e-805d-7c35c93eb04b-maas-api-xks-ca" pod "103a57d0-2aca-476e-805d-7c35c93eb04b" (UID: "103a57d0-2aca-476e-805d-7c35c93eb04b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:59.198485 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:59.198411 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103a57d0-2aca-476e-805d-7c35c93eb04b-kube-api-access-9jkzl" pod "103a57d0-2aca-476e-805d-7c35c93eb04b" (UID: "103a57d0-2aca-476e-805d-7c35c93eb04b"). InnerVolumeSpecName "kube-api-access-9jkzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:59.297539 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:59.297510 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103a57d0-2aca-476e-805d-7c35c93eb04b-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:59.297705 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:59.297542 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jkzl\" (UniqueName: \"kubernetes.io/projected/103a57d0-2aca-476e-805d-7c35c93eb04b-kube-api-access-9jkzl\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:59.931123 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:59.931093 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c" Oct 30 22:24:59.931528 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:59.931090 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c" event={"ID":"103a57d0-2aca-476e-805d-7c35c93eb04b","Type":"ContainerDied","Data":"4de60ae6bdd2414699051a8eeb897ddabff0fd42ccafe1f643f4e40a062eb9d9"} Oct 30 22:24:59.931528 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:59.931232 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4de60ae6bdd2414699051a8eeb897ddabff0fd42ccafe1f643f4e40a062eb9d9" Oct 30 22:24:59.956858 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:59.956828 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" containerID="cri-o://1701349ffec22d1b0ac48a1e52313d7fdfc0cb4d3bb6956f8ab2d299a4486f03" gracePeriod=45 Oct 30 22:24:59.969392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:24:59.969363 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" containerID="cri-o://538a85c8c90c8cb32af30f0605ed250b624501b0e272e35189f8c48c4cbca517" gracePeriod=45 Oct 30 22:25:00.276898 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.276876 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:25:00.280756 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.280735 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:25:00.409161 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.409133 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-home\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-home\") pod \"6173209e-9700-444d-b33a-cabb2d51816a\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " Oct 30 22:25:00.409344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.409179 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-home\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-home\") pod \"fb2bc88a-bb73-4e24-9dce-252844253f65\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " Oct 30 22:25:00.409344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.409218 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6173209e-9700-444d-b33a-cabb2d51816a-kube-api-access-b6qcz\" (UniqueName: \"kubernetes.io/projected/6173209e-9700-444d-b33a-cabb2d51816a-kube-api-access-b6qcz\") pod \"6173209e-9700-444d-b33a-cabb2d51816a\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " Oct 30 22:25:00.409344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.409302 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-dshm\") pod \"6173209e-9700-444d-b33a-cabb2d51816a\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " Oct 30 22:25:00.409514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.409347 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-dshm\") pod \"fb2bc88a-bb73-4e24-9dce-252844253f65\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " Oct 30 22:25:00.409514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.409375 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-tmp-dir\") pod \"fb2bc88a-bb73-4e24-9dce-252844253f65\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " Oct 30 22:25:00.409514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.409419 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb2bc88a-bb73-4e24-9dce-252844253f65-kube-api-access-ngsdr\" (UniqueName: \"kubernetes.io/projected/fb2bc88a-bb73-4e24-9dce-252844253f65-kube-api-access-ngsdr\") pod \"fb2bc88a-bb73-4e24-9dce-252844253f65\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " Oct 30 22:25:00.409514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.409432 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-home" pod "fb2bc88a-bb73-4e24-9dce-252844253f65" (UID: "fb2bc88a-bb73-4e24-9dce-252844253f65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:00.409514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.409501 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb2bc88a-bb73-4e24-9dce-252844253f65-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb2bc88a-bb73-4e24-9dce-252844253f65-tls-certs\") pod \"fb2bc88a-bb73-4e24-9dce-252844253f65\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " Oct 30 22:25:00.410491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.409633 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-tmp-dir\") pod \"6173209e-9700-444d-b33a-cabb2d51816a\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " Oct 30 22:25:00.410491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.409666 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6173209e-9700-444d-b33a-cabb2d51816a-tls-certs\" (UniqueName: \"kubernetes.io/secret/6173209e-9700-444d-b33a-cabb2d51816a-tls-certs\") pod \"6173209e-9700-444d-b33a-cabb2d51816a\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " Oct 30 22:25:00.410491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.409673 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-tmp-dir" pod "fb2bc88a-bb73-4e24-9dce-252844253f65" (UID: "fb2bc88a-bb73-4e24-9dce-252844253f65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:00.410491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.409722 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-model-cache\") pod \"fb2bc88a-bb73-4e24-9dce-252844253f65\" (UID: \"fb2bc88a-bb73-4e24-9dce-252844253f65\") " Oct 30 22:25:00.410491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.409728 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-home" pod "6173209e-9700-444d-b33a-cabb2d51816a" (UID: "6173209e-9700-444d-b33a-cabb2d51816a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:00.410491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.409787 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-model-cache\") pod \"6173209e-9700-444d-b33a-cabb2d51816a\" (UID: \"6173209e-9700-444d-b33a-cabb2d51816a\") " Oct 30 22:25:00.410491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.409880 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-tmp-dir" pod "6173209e-9700-444d-b33a-cabb2d51816a" (UID: "6173209e-9700-444d-b33a-cabb2d51816a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:00.410491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.409906 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-model-cache" pod "fb2bc88a-bb73-4e24-9dce-252844253f65" (UID: "fb2bc88a-bb73-4e24-9dce-252844253f65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:00.410491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.410154 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-model-cache" pod "6173209e-9700-444d-b33a-cabb2d51816a" (UID: "6173209e-9700-444d-b33a-cabb2d51816a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:00.411180 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.410601 4335 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-tmp-dir\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:00.411180 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.410645 4335 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-model-cache\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:00.411180 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.410659 4335 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-model-cache\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:00.411180 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.410672 4335 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-home\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:00.411180 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.410686 4335 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-home\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:00.411180 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.410698 4335 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-tmp-dir\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:00.412943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.412910 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6173209e-9700-444d-b33a-cabb2d51816a-tls-certs" pod "6173209e-9700-444d-b33a-cabb2d51816a" (UID: "6173209e-9700-444d-b33a-cabb2d51816a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:00.413065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.413037 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6173209e-9700-444d-b33a-cabb2d51816a-kube-api-access-b6qcz" pod "6173209e-9700-444d-b33a-cabb2d51816a" (UID: "6173209e-9700-444d-b33a-cabb2d51816a"). InnerVolumeSpecName "kube-api-access-b6qcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:00.413190 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.413166 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb2bc88a-bb73-4e24-9dce-252844253f65-tls-certs" pod "fb2bc88a-bb73-4e24-9dce-252844253f65" (UID: "fb2bc88a-bb73-4e24-9dce-252844253f65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:00.414006 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.413977 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2bc88a-bb73-4e24-9dce-252844253f65-kube-api-access-ngsdr" pod "fb2bc88a-bb73-4e24-9dce-252844253f65" (UID: "fb2bc88a-bb73-4e24-9dce-252844253f65"). InnerVolumeSpecName "kube-api-access-ngsdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:00.414097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.414027 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-dshm" pod "6173209e-9700-444d-b33a-cabb2d51816a" (UID: "6173209e-9700-444d-b33a-cabb2d51816a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:00.414463 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.414445 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-dshm" pod "fb2bc88a-bb73-4e24-9dce-252844253f65" (UID: "fb2bc88a-bb73-4e24-9dce-252844253f65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:00.513342 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.513248 4335 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6173209e-9700-444d-b33a-cabb2d51816a-dshm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:00.513342 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.513291 4335 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb2bc88a-bb73-4e24-9dce-252844253f65-dshm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:00.513342 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.513309 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngsdr\" (UniqueName: \"kubernetes.io/projected/fb2bc88a-bb73-4e24-9dce-252844253f65-kube-api-access-ngsdr\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:00.513342 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.513324 4335 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb2bc88a-bb73-4e24-9dce-252844253f65-tls-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:00.513342 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.513339 4335 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6173209e-9700-444d-b33a-cabb2d51816a-tls-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:00.513831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.513380 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6qcz\" (UniqueName: \"kubernetes.io/projected/6173209e-9700-444d-b33a-cabb2d51816a-kube-api-access-b6qcz\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:00.941368 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.941333 4335 generic.go:356] "Generic (PLEG): container finished" podID="6173209e-9700-444d-b33a-cabb2d51816a" containerID="1701349ffec22d1b0ac48a1e52313d7fdfc0cb4d3bb6956f8ab2d299a4486f03" exitCode=0 Oct 30 22:25:00.941908 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.941381 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" Oct 30 22:25:00.941908 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.941413 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" event={"ID":"6173209e-9700-444d-b33a-cabb2d51816a","Type":"ContainerDied","Data":"1701349ffec22d1b0ac48a1e52313d7fdfc0cb4d3bb6956f8ab2d299a4486f03"} Oct 30 22:25:00.941908 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.941468 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn" event={"ID":"6173209e-9700-444d-b33a-cabb2d51816a","Type":"ContainerDied","Data":"f9860d77b43777d8e96ac669b1ff880153b2ede7a5f718c43f43986cfd3f6da0"} Oct 30 22:25:00.941908 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.941492 4335 scope.go:122] "RemoveContainer" containerID="1701349ffec22d1b0ac48a1e52313d7fdfc0cb4d3bb6956f8ab2d299a4486f03" Oct 30 22:25:00.943251 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.943229 4335 generic.go:356] "Generic (PLEG): container finished" podID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerID="538a85c8c90c8cb32af30f0605ed250b624501b0e272e35189f8c48c4cbca517" exitCode=0 Oct 30 22:25:00.943373 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.943269 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" Oct 30 22:25:00.943373 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.943319 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" event={"ID":"fb2bc88a-bb73-4e24-9dce-252844253f65","Type":"ContainerDied","Data":"538a85c8c90c8cb32af30f0605ed250b624501b0e272e35189f8c48c4cbca517"} Oct 30 22:25:00.943373 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.943355 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b" event={"ID":"fb2bc88a-bb73-4e24-9dce-252844253f65","Type":"ContainerDied","Data":"bc12c24f6d917c52ad91dec2444c9478022982d8c4c97b4bdc9941d8ef85468f"} Oct 30 22:25:00.954059 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.954043 4335 scope.go:122] "RemoveContainer" containerID="1701349ffec22d1b0ac48a1e52313d7fdfc0cb4d3bb6956f8ab2d299a4486f03" Oct 30 22:25:00.954308 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:25:00.954291 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1701349ffec22d1b0ac48a1e52313d7fdfc0cb4d3bb6956f8ab2d299a4486f03\": container with ID starting with 1701349ffec22d1b0ac48a1e52313d7fdfc0cb4d3bb6956f8ab2d299a4486f03 not found: ID does not exist" containerID="1701349ffec22d1b0ac48a1e52313d7fdfc0cb4d3bb6956f8ab2d299a4486f03" Oct 30 22:25:00.954356 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.954314 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1701349ffec22d1b0ac48a1e52313d7fdfc0cb4d3bb6956f8ab2d299a4486f03"} err="failed to get container status \"1701349ffec22d1b0ac48a1e52313d7fdfc0cb4d3bb6956f8ab2d299a4486f03\": rpc error: code = NotFound desc = could not find container \"1701349ffec22d1b0ac48a1e52313d7fdfc0cb4d3bb6956f8ab2d299a4486f03\": container with ID starting with 1701349ffec22d1b0ac48a1e52313d7fdfc0cb4d3bb6956f8ab2d299a4486f03 not found: ID does not exist" Oct 30 22:25:00.954356 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.954331 4335 scope.go:122] "RemoveContainer" containerID="538a85c8c90c8cb32af30f0605ed250b624501b0e272e35189f8c48c4cbca517" Oct 30 22:25:00.965248 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.965231 4335 scope.go:122] "RemoveContainer" containerID="538a85c8c90c8cb32af30f0605ed250b624501b0e272e35189f8c48c4cbca517" Oct 30 22:25:00.965503 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:25:00.965486 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538a85c8c90c8cb32af30f0605ed250b624501b0e272e35189f8c48c4cbca517\": container with ID starting with 538a85c8c90c8cb32af30f0605ed250b624501b0e272e35189f8c48c4cbca517 not found: ID does not exist" containerID="538a85c8c90c8cb32af30f0605ed250b624501b0e272e35189f8c48c4cbca517" Oct 30 22:25:00.965552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:00.965508 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538a85c8c90c8cb32af30f0605ed250b624501b0e272e35189f8c48c4cbca517"} err="failed to get container status \"538a85c8c90c8cb32af30f0605ed250b624501b0e272e35189f8c48c4cbca517\": rpc error: code = NotFound desc = could not find container \"538a85c8c90c8cb32af30f0605ed250b624501b0e272e35189f8c48c4cbca517\": container with ID starting with 538a85c8c90c8cb32af30f0605ed250b624501b0e272e35189f8c48c4cbca517 not found: ID does not exist" Oct 30 22:25:01.110240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:01.110195 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn"] Oct 30 22:25:01.116839 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:01.116811 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-bf7cbf/distinct-2-w1-bf7cbf-kserve-7c95687f45-c7hsn"] Oct 30 22:25:01.222147 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:01.222074 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b"] Oct 30 22:25:01.226273 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:01.226249 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-bf7cbf/distinct-w1-bf7cbf-kserve-74fb5d48b-7nd5b"] Oct 30 22:25:03.064328 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.064284 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6173209e-9700-444d-b33a-cabb2d51816a" path="/var/lib/kubelet/pods/6173209e-9700-444d-b33a-cabb2d51816a/volumes" Oct 30 22:25:03.064841 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.064825 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" path="/var/lib/kubelet/pods/fb2bc88a-bb73-4e24-9dce-252844253f65/volumes" Oct 30 22:25:03.144430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.144398 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb"] Oct 30 22:25:03.145412 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.145391 4335 cpu_manager.go:414] "containerMap: removing container" podUID="103a57d0-2aca-476e-805d-7c35c93eb04b" containerName="revoke-keys" Oct 30 22:25:03.145488 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.145417 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103a57d0-2aca-476e-805d-7c35c93eb04b" containerName="revoke-keys" Oct 30 22:25:03.145488 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.145439 4335 cpu_manager.go:414] "containerMap: removing container" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" Oct 30 22:25:03.145488 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.145447 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" Oct 30 22:25:03.145488 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.145459 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8" containerName="revoke-keys" Oct 30 22:25:03.145488 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.145467 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8" containerName="revoke-keys" Oct 30 22:25:03.145488 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.145482 4335 cpu_manager.go:414] "containerMap: removing container" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" Oct 30 22:25:03.145684 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.145490 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" Oct 30 22:25:03.145725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.145708 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="103a57d0-2aca-476e-805d-7c35c93eb04b" containerName="revoke-keys" Oct 30 22:25:03.145788 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.145728 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb2bc88a-bb73-4e24-9dce-252844253f65" containerName="main" Oct 30 22:25:03.145788 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.145738 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8" containerName="revoke-keys" Oct 30 22:25:03.145788 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.145748 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="6173209e-9700-444d-b33a-cabb2d51816a" containerName="main" Oct 30 22:25:03.151563 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.151543 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" Oct 30 22:25:03.203527 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.203505 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-026c11\"" Oct 30 22:25:03.203684 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.203529 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-026c11-dockercfg-cttlc\"" Oct 30 22:25:03.313582 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.313537 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpm8j\" (UniqueName: \"kubernetes.io/projected/a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc-kube-api-access-cpm8j\") pod \"payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb\" (UID: \"a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" Oct 30 22:25:03.313956 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.313926 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb\" (UID: \"a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" Oct 30 22:25:03.327808 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.327741 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb"] Oct 30 22:25:03.344176 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.344136 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn"] Oct 30 22:25:03.358954 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.358916 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" Oct 30 22:25:03.364250 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.364224 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn"] Oct 30 22:25:03.415207 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.415176 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpm8j\" (UniqueName: \"kubernetes.io/projected/a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc-kube-api-access-cpm8j\") pod \"payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb\" (UID: \"a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" Oct 30 22:25:03.415353 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.415238 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb\" (UID: \"a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" Oct 30 22:25:03.416104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.416077 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb\" (UID: \"a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" Oct 30 22:25:03.476967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.476935 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpm8j\" (UniqueName: \"kubernetes.io/projected/a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc-kube-api-access-cpm8j\") pod \"payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb\" (UID: \"a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" Oct 30 22:25:03.516355 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.516329 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f63b65f-f924-4f71-afc5-3a5418226ffa-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn\" (UID: \"7f63b65f-f924-4f71-afc5-3a5418226ffa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" Oct 30 22:25:03.516464 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.516384 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trv6q\" (UniqueName: \"kubernetes.io/projected/7f63b65f-f924-4f71-afc5-3a5418226ffa-kube-api-access-trv6q\") pod \"payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn\" (UID: \"7f63b65f-f924-4f71-afc5-3a5418226ffa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" Oct 30 22:25:03.617777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.617709 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f63b65f-f924-4f71-afc5-3a5418226ffa-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn\" (UID: \"7f63b65f-f924-4f71-afc5-3a5418226ffa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" Oct 30 22:25:03.617902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.617793 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trv6q\" (UniqueName: \"kubernetes.io/projected/7f63b65f-f924-4f71-afc5-3a5418226ffa-kube-api-access-trv6q\") pod \"payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn\" (UID: \"7f63b65f-f924-4f71-afc5-3a5418226ffa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" Oct 30 22:25:03.618370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.618349 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f63b65f-f924-4f71-afc5-3a5418226ffa-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn\" (UID: \"7f63b65f-f924-4f71-afc5-3a5418226ffa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" Oct 30 22:25:03.679064 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.679032 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trv6q\" (UniqueName: \"kubernetes.io/projected/7f63b65f-f924-4f71-afc5-3a5418226ffa-kube-api-access-trv6q\") pod \"payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn\" (UID: \"7f63b65f-f924-4f71-afc5-3a5418226ffa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" Oct 30 22:25:03.769876 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.769850 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" Oct 30 22:25:03.870285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.869880 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7"] Oct 30 22:25:03.983870 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.983840 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" Oct 30 22:25:03.986274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:03.986246 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp"] Oct 30 22:25:04.154128 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:25:04.154097 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d790c0_2e37_43f7_ad4a_1a5c8b2637bc.slice/crio-b970fb011c2ff49915ae4a9a678fae6e4fe2e94cb44511154909f60d8bfffc1e WatchSource:0}: Error finding container b970fb011c2ff49915ae4a9a678fae6e4fe2e94cb44511154909f60d8bfffc1e: Status 404 returned error can't find the container with id b970fb011c2ff49915ae4a9a678fae6e4fe2e94cb44511154909f60d8bfffc1e Oct 30 22:25:04.156783 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:04.156754 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb"] Oct 30 22:25:04.374340 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:25:04.374306 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f63b65f_f924_4f71_afc5_3a5418226ffa.slice/crio-480a0a45fc62620c3863e727563e1a0277cba87076ea224fa1376c0cfb961639 WatchSource:0}: Error finding container 480a0a45fc62620c3863e727563e1a0277cba87076ea224fa1376c0cfb961639: Status 404 returned error can't find the container with id 480a0a45fc62620c3863e727563e1a0277cba87076ea224fa1376c0cfb961639 Oct 30 22:25:04.375273 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:04.375230 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn"] Oct 30 22:25:04.982317 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:04.982278 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" event={"ID":"a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc","Type":"ContainerStarted","Data":"727be62bb9254a13fc445a4e23d9486e4efc46010c82cee5e8c29362216c6d5c"} Oct 30 22:25:04.982495 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:04.982343 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" Oct 30 22:25:04.982495 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:04.982364 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" event={"ID":"a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc","Type":"ContainerStarted","Data":"b970fb011c2ff49915ae4a9a678fae6e4fe2e94cb44511154909f60d8bfffc1e"} Oct 30 22:25:04.984170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:04.984132 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" event={"ID":"7f63b65f-f924-4f71-afc5-3a5418226ffa","Type":"ContainerStarted","Data":"acf6f6bb3d488e338e68d12861556465d1024cfa6a02a95009773f868217522e"} Oct 30 22:25:04.984274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:04.984184 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" event={"ID":"7f63b65f-f924-4f71-afc5-3a5418226ffa","Type":"ContainerStarted","Data":"480a0a45fc62620c3863e727563e1a0277cba87076ea224fa1376c0cfb961639"} Oct 30 22:25:04.984274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:04.984215 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" Oct 30 22:25:05.098525 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:05.098476 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" podStartSLOduration=2.098462002 podStartE2EDuration="2.098462002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:05.096970121 +0000 UTC m=+2438.197401855" watchObservedRunningTime="2026-10-30 22:25:05.098462002 +0000 UTC m=+2438.198893733" Oct 30 22:25:05.223077 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:05.222767 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" podStartSLOduration=2.22274834 podStartE2EDuration="2.22274834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:05.219363268 +0000 UTC m=+2438.319795007" watchObservedRunningTime="2026-10-30 22:25:05.22274834 +0000 UTC m=+2438.323180074" Oct 30 22:25:06.465080 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:06.465035 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7"] Oct 30 22:25:07.157748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:07.157703 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" podUID="dd65776e-f29f-436a-aabb-e9e6b6d3497a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:25:08.050390 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:08.050359 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5"] Oct 30 22:25:08.099214 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:08.099185 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5feaf50-cf2v5"] Oct 30 22:25:08.870591 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:08.870553 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7" podUID="c1388034-f774-4af1-bdb7-fa9eb1ec07f5" containerName="payload-processing" containerID="cri-o://48f8fdc4fd20faf79bfbaec185554586591cc4f6e41c92053422f6d26472da66" gracePeriod=25 Oct 30 22:25:08.987287 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:08.987252 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp" podUID="472bc81d-c1a3-463d-98fe-b98d4576aab7" containerName="payload-pre-processing" containerID="cri-o://c1d02859a0c542c1d3197585f126fdf0b8b12586c52af23eeaf9527fd35c4120" gracePeriod=25 Oct 30 22:25:09.024388 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.024355 4335 generic.go:356] "Generic (PLEG): container finished" podID="c1388034-f774-4af1-bdb7-fa9eb1ec07f5" containerID="48f8fdc4fd20faf79bfbaec185554586591cc4f6e41c92053422f6d26472da66" exitCode=0 Oct 30 22:25:09.024526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.024435 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7" event={"ID":"c1388034-f774-4af1-bdb7-fa9eb1ec07f5","Type":"ContainerDied","Data":"48f8fdc4fd20faf79bfbaec185554586591cc4f6e41c92053422f6d26472da66"} Oct 30 22:25:09.058980 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.058954 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26693c1f-2409-4b8b-94ed-c52e46123c3e" path="/var/lib/kubelet/pods/26693c1f-2409-4b8b-94ed-c52e46123c3e/volumes" Oct 30 22:25:09.187420 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.187397 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7" Oct 30 22:25:09.306423 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.306075 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp" Oct 30 22:25:09.314782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.314742 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1388034-f774-4af1-bdb7-fa9eb1ec07f5-kube-api-access-dddr7\" (UniqueName: \"kubernetes.io/projected/c1388034-f774-4af1-bdb7-fa9eb1ec07f5-kube-api-access-dddr7\") pod \"c1388034-f774-4af1-bdb7-fa9eb1ec07f5\" (UID: \"c1388034-f774-4af1-bdb7-fa9eb1ec07f5\") " Oct 30 22:25:09.314897 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.314793 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1388034-f774-4af1-bdb7-fa9eb1ec07f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1388034-f774-4af1-bdb7-fa9eb1ec07f5-plugins-config-volume\") pod \"c1388034-f774-4af1-bdb7-fa9eb1ec07f5\" (UID: \"c1388034-f774-4af1-bdb7-fa9eb1ec07f5\") " Oct 30 22:25:09.315444 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.315420 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1388034-f774-4af1-bdb7-fa9eb1ec07f5-plugins-config-volume" pod "c1388034-f774-4af1-bdb7-fa9eb1ec07f5" (UID: "c1388034-f774-4af1-bdb7-fa9eb1ec07f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:09.317556 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.317533 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1388034-f774-4af1-bdb7-fa9eb1ec07f5-kube-api-access-dddr7" pod "c1388034-f774-4af1-bdb7-fa9eb1ec07f5" (UID: "c1388034-f774-4af1-bdb7-fa9eb1ec07f5"). InnerVolumeSpecName "kube-api-access-dddr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:09.416020 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.415960 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/472bc81d-c1a3-463d-98fe-b98d4576aab7-kube-api-access-p275l\" (UniqueName: \"kubernetes.io/projected/472bc81d-c1a3-463d-98fe-b98d4576aab7-kube-api-access-p275l\") pod \"472bc81d-c1a3-463d-98fe-b98d4576aab7\" (UID: \"472bc81d-c1a3-463d-98fe-b98d4576aab7\") " Oct 30 22:25:09.416168 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.416149 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/472bc81d-c1a3-463d-98fe-b98d4576aab7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472bc81d-c1a3-463d-98fe-b98d4576aab7-plugins-config-volume\") pod \"472bc81d-c1a3-463d-98fe-b98d4576aab7\" (UID: \"472bc81d-c1a3-463d-98fe-b98d4576aab7\") " Oct 30 22:25:09.416714 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.416686 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472bc81d-c1a3-463d-98fe-b98d4576aab7-plugins-config-volume" pod "472bc81d-c1a3-463d-98fe-b98d4576aab7" (UID: "472bc81d-c1a3-463d-98fe-b98d4576aab7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:09.416818 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.416753 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dddr7\" (UniqueName: \"kubernetes.io/projected/c1388034-f774-4af1-bdb7-fa9eb1ec07f5-kube-api-access-dddr7\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:09.416818 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.416771 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1388034-f774-4af1-bdb7-fa9eb1ec07f5-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:09.418731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.418707 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472bc81d-c1a3-463d-98fe-b98d4576aab7-kube-api-access-p275l" pod "472bc81d-c1a3-463d-98fe-b98d4576aab7" (UID: "472bc81d-c1a3-463d-98fe-b98d4576aab7"). InnerVolumeSpecName "kube-api-access-p275l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:09.517811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.517780 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p275l\" (UniqueName: \"kubernetes.io/projected/472bc81d-c1a3-463d-98fe-b98d4576aab7-kube-api-access-p275l\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:09.517811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.517805 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472bc81d-c1a3-463d-98fe-b98d4576aab7-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:09.951077 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.951046 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w"] Oct 30 22:25:09.955440 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:09.955417 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0f3344-z4w4w"] Oct 30 22:25:10.036078 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:10.036053 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7" Oct 30 22:25:10.036227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:10.036073 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7" event={"ID":"c1388034-f774-4af1-bdb7-fa9eb1ec07f5","Type":"ContainerDied","Data":"0f658c45238180561ff75d3afe0b0380b7edb0463eb52fd4485dad77059445a9"} Oct 30 22:25:10.036227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:10.036129 4335 scope.go:122] "RemoveContainer" containerID="48f8fdc4fd20faf79bfbaec185554586591cc4f6e41c92053422f6d26472da66" Oct 30 22:25:10.037837 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:10.037813 4335 generic.go:356] "Generic (PLEG): container finished" podID="472bc81d-c1a3-463d-98fe-b98d4576aab7" containerID="c1d02859a0c542c1d3197585f126fdf0b8b12586c52af23eeaf9527fd35c4120" exitCode=0 Oct 30 22:25:10.037988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:10.037843 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp" event={"ID":"472bc81d-c1a3-463d-98fe-b98d4576aab7","Type":"ContainerDied","Data":"c1d02859a0c542c1d3197585f126fdf0b8b12586c52af23eeaf9527fd35c4120"} Oct 30 22:25:10.038121 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:10.038014 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp" event={"ID":"472bc81d-c1a3-463d-98fe-b98d4576aab7","Type":"ContainerDied","Data":"486d45ffeb5fcfbb73121a81f89d7e926507ed6134325577c5ba04fc730d3049"} Oct 30 22:25:10.038121 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:10.037867 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp" Oct 30 22:25:10.048270 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:10.048250 4335 scope.go:122] "RemoveContainer" containerID="c1d02859a0c542c1d3197585f126fdf0b8b12586c52af23eeaf9527fd35c4120" Oct 30 22:25:10.058762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:10.058746 4335 scope.go:122] "RemoveContainer" containerID="c1d02859a0c542c1d3197585f126fdf0b8b12586c52af23eeaf9527fd35c4120" Oct 30 22:25:10.058997 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:25:10.058980 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d02859a0c542c1d3197585f126fdf0b8b12586c52af23eeaf9527fd35c4120\": container with ID starting with c1d02859a0c542c1d3197585f126fdf0b8b12586c52af23eeaf9527fd35c4120 not found: ID does not exist" containerID="c1d02859a0c542c1d3197585f126fdf0b8b12586c52af23eeaf9527fd35c4120" Oct 30 22:25:10.059294 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:10.059002 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d02859a0c542c1d3197585f126fdf0b8b12586c52af23eeaf9527fd35c4120"} err="failed to get container status \"c1d02859a0c542c1d3197585f126fdf0b8b12586c52af23eeaf9527fd35c4120\": rpc error: code = NotFound desc = could not find container \"c1d02859a0c542c1d3197585f126fdf0b8b12586c52af23eeaf9527fd35c4120\": container with ID starting with c1d02859a0c542c1d3197585f126fdf0b8b12586c52af23eeaf9527fd35c4120 not found: ID does not exist" Oct 30 22:25:10.202598 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:10.202530 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7"] Oct 30 22:25:10.206756 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:10.206733 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25237a-fbc65666c-hxbf7"] Oct 30 22:25:10.317328 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:10.317289 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp"] Oct 30 22:25:10.325243 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:10.325204 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25237a-7d75989df5-q9tvp"] Oct 30 22:25:11.063257 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:11.063226 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="472bc81d-c1a3-463d-98fe-b98d4576aab7" path="/var/lib/kubelet/pods/472bc81d-c1a3-463d-98fe-b98d4576aab7/volumes" Oct 30 22:25:11.063688 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:11.063603 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1388034-f774-4af1-bdb7-fa9eb1ec07f5" path="/var/lib/kubelet/pods/c1388034-f774-4af1-bdb7-fa9eb1ec07f5/volumes" Oct 30 22:25:11.064012 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:11.063997 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e207549a-e5fa-43e5-96e1-45402d3aa40b" path="/var/lib/kubelet/pods/e207549a-e5fa-43e5-96e1-45402d3aa40b/volumes" Oct 30 22:25:11.466242 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:11.466162 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7" podUID="8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1" containerName="payload-pre-processing" containerID="cri-o://f26c219bc75ac5ad8acc006319fff4b0a6e5e0708fe481a64e6a18f767a72bd6" gracePeriod=25 Oct 30 22:25:11.760498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:11.760470 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7" Oct 30 22:25:11.849223 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:11.849186 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1-plugins-config-volume\") pod \"8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1\" (UID: \"8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1\") " Oct 30 22:25:11.849421 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:11.849403 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1-kube-api-access-lhhhn\" (UniqueName: \"kubernetes.io/projected/8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1-kube-api-access-lhhhn\") pod \"8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1\" (UID: \"8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1\") " Oct 30 22:25:11.849566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:11.849537 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1-plugins-config-volume" pod "8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1" (UID: "8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:11.849937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:11.849915 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:11.851676 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:11.851653 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1-kube-api-access-lhhhn" pod "8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1" (UID: "8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1"). InnerVolumeSpecName "kube-api-access-lhhhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:11.951071 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:11.951040 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhhhn\" (UniqueName: \"kubernetes.io/projected/8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1-kube-api-access-lhhhn\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:12.068857 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:12.068828 4335 generic.go:356] "Generic (PLEG): container finished" podID="8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1" containerID="f26c219bc75ac5ad8acc006319fff4b0a6e5e0708fe481a64e6a18f767a72bd6" exitCode=0 Oct 30 22:25:12.069281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:12.068872 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7" Oct 30 22:25:12.069281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:12.068895 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7" event={"ID":"8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1","Type":"ContainerDied","Data":"f26c219bc75ac5ad8acc006319fff4b0a6e5e0708fe481a64e6a18f767a72bd6"} Oct 30 22:25:12.069281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:12.068950 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7" event={"ID":"8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1","Type":"ContainerDied","Data":"7cb0ea0a7fa6ff7af10ee6072e8fe33c5b0caf786ac61f261f7e8b18a4f4eb55"} Oct 30 22:25:12.069281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:12.068981 4335 scope.go:122] "RemoveContainer" containerID="f26c219bc75ac5ad8acc006319fff4b0a6e5e0708fe481a64e6a18f767a72bd6" Oct 30 22:25:12.080487 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:12.080472 4335 scope.go:122] "RemoveContainer" containerID="f26c219bc75ac5ad8acc006319fff4b0a6e5e0708fe481a64e6a18f767a72bd6" Oct 30 22:25:12.080792 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:25:12.080771 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f26c219bc75ac5ad8acc006319fff4b0a6e5e0708fe481a64e6a18f767a72bd6\": container with ID starting with f26c219bc75ac5ad8acc006319fff4b0a6e5e0708fe481a64e6a18f767a72bd6 not found: ID does not exist" containerID="f26c219bc75ac5ad8acc006319fff4b0a6e5e0708fe481a64e6a18f767a72bd6" Oct 30 22:25:12.080842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:12.080799 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26c219bc75ac5ad8acc006319fff4b0a6e5e0708fe481a64e6a18f767a72bd6"} err="failed to get container status \"f26c219bc75ac5ad8acc006319fff4b0a6e5e0708fe481a64e6a18f767a72bd6\": rpc error: code = NotFound desc = could not find container \"f26c219bc75ac5ad8acc006319fff4b0a6e5e0708fe481a64e6a18f767a72bd6\": container with ID starting with f26c219bc75ac5ad8acc006319fff4b0a6e5e0708fe481a64e6a18f767a72bd6 not found: ID does not exist" Oct 30 22:25:12.235212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:12.235180 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7"] Oct 30 22:25:12.239729 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:12.239706 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b8a091-758674f567-6znv7"] Oct 30 22:25:13.059274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:13.059242 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1" path="/var/lib/kubelet/pods/8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1/volumes" Oct 30 22:25:15.997244 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:15.997214 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" Oct 30 22:25:15.998219 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:15.998197 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" Oct 30 22:25:16.497318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:16.497284 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c"] Oct 30 22:25:16.503928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:16.503903 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b8a091-lsz7c"] Oct 30 22:25:16.768430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:16.768406 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" Oct 30 22:25:16.916633 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:16.916585 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd65776e-f29f-436a-aabb-e9e6b6d3497a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd65776e-f29f-436a-aabb-e9e6b6d3497a-plugins-config-volume\") pod \"dd65776e-f29f-436a-aabb-e9e6b6d3497a\" (UID: \"dd65776e-f29f-436a-aabb-e9e6b6d3497a\") " Oct 30 22:25:16.916868 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:16.916662 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd65776e-f29f-436a-aabb-e9e6b6d3497a-kube-api-access-bdqs9\" (UniqueName: \"kubernetes.io/projected/dd65776e-f29f-436a-aabb-e9e6b6d3497a-kube-api-access-bdqs9\") pod \"dd65776e-f29f-436a-aabb-e9e6b6d3497a\" (UID: \"dd65776e-f29f-436a-aabb-e9e6b6d3497a\") " Oct 30 22:25:16.917021 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:16.916997 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd65776e-f29f-436a-aabb-e9e6b6d3497a-plugins-config-volume" pod "dd65776e-f29f-436a-aabb-e9e6b6d3497a" (UID: "dd65776e-f29f-436a-aabb-e9e6b6d3497a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:16.919050 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:16.919024 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd65776e-f29f-436a-aabb-e9e6b6d3497a-kube-api-access-bdqs9" pod "dd65776e-f29f-436a-aabb-e9e6b6d3497a" (UID: "dd65776e-f29f-436a-aabb-e9e6b6d3497a"). InnerVolumeSpecName "kube-api-access-bdqs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:17.018267 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:17.018185 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd65776e-f29f-436a-aabb-e9e6b6d3497a-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:17.018267 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:17.018222 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdqs9\" (UniqueName: \"kubernetes.io/projected/dd65776e-f29f-436a-aabb-e9e6b6d3497a-kube-api-access-bdqs9\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:17.058524 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:17.058498 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103a57d0-2aca-476e-805d-7c35c93eb04b" path="/var/lib/kubelet/pods/103a57d0-2aca-476e-805d-7c35c93eb04b/volumes" Oct 30 22:25:17.123695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:17.123664 4335 generic.go:356] "Generic (PLEG): container finished" podID="dd65776e-f29f-436a-aabb-e9e6b6d3497a" containerID="3c8b283c688e0a53da0918edb87abe4a24c6323b112bc52e49f7e8a7496267b0" exitCode=0 Oct 30 22:25:17.123792 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:17.123743 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" Oct 30 22:25:17.123846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:17.123790 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" event={"ID":"dd65776e-f29f-436a-aabb-e9e6b6d3497a","Type":"ContainerDied","Data":"3c8b283c688e0a53da0918edb87abe4a24c6323b112bc52e49f7e8a7496267b0"} Oct 30 22:25:17.123882 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:17.123846 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b" event={"ID":"dd65776e-f29f-436a-aabb-e9e6b6d3497a","Type":"ContainerDied","Data":"74d7c26e3b0026844dea7aaa801428e1ef06d17e516740c37a78a329dea345f1"} Oct 30 22:25:17.123882 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:17.123877 4335 scope.go:122] "RemoveContainer" containerID="3c8b283c688e0a53da0918edb87abe4a24c6323b112bc52e49f7e8a7496267b0" Oct 30 22:25:17.136628 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:17.136593 4335 scope.go:122] "RemoveContainer" containerID="3c8b283c688e0a53da0918edb87abe4a24c6323b112bc52e49f7e8a7496267b0" Oct 30 22:25:17.136929 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:25:17.136910 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8b283c688e0a53da0918edb87abe4a24c6323b112bc52e49f7e8a7496267b0\": container with ID starting with 3c8b283c688e0a53da0918edb87abe4a24c6323b112bc52e49f7e8a7496267b0 not found: ID does not exist" containerID="3c8b283c688e0a53da0918edb87abe4a24c6323b112bc52e49f7e8a7496267b0" Oct 30 22:25:17.136994 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:17.136937 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8b283c688e0a53da0918edb87abe4a24c6323b112bc52e49f7e8a7496267b0"} err="failed to get container status \"3c8b283c688e0a53da0918edb87abe4a24c6323b112bc52e49f7e8a7496267b0\": rpc error: code = NotFound desc = could not find container \"3c8b283c688e0a53da0918edb87abe4a24c6323b112bc52e49f7e8a7496267b0\": container with ID starting with 3c8b283c688e0a53da0918edb87abe4a24c6323b112bc52e49f7e8a7496267b0 not found: ID does not exist" Oct 30 22:25:17.291395 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:17.291327 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b"] Oct 30 22:25:17.340085 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:17.340049 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-bf7cbf-55459d865-6dp9b"] Oct 30 22:25:19.061825 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:19.061794 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd65776e-f29f-436a-aabb-e9e6b6d3497a" path="/var/lib/kubelet/pods/dd65776e-f29f-436a-aabb-e9e6b6d3497a/volumes" Oct 30 22:25:19.604405 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:19.604371 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:25:25.321109 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:25:25.321071 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:25:33.549424 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:33.549392 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg"] Oct 30 22:25:33.598525 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:33.598496 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-2b7317-mgwlg"] Oct 30 22:25:35.059935 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:35.059902 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c97c3b-4a65-4885-9d2e-f67fd1336947" path="/var/lib/kubelet/pods/c3c97c3b-4a65-4885-9d2e-f67fd1336947/volumes" Oct 30 22:25:36.254778 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.254743 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg"] Oct 30 22:25:36.255368 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.255354 4335 cpu_manager.go:414] "containerMap: removing container" podUID="c1388034-f774-4af1-bdb7-fa9eb1ec07f5" containerName="payload-processing" Oct 30 22:25:36.255414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.255369 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1388034-f774-4af1-bdb7-fa9eb1ec07f5" containerName="payload-processing" Oct 30 22:25:36.255414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.255388 4335 cpu_manager.go:414] "containerMap: removing container" podUID="8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1" containerName="payload-pre-processing" Oct 30 22:25:36.255414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.255393 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1" containerName="payload-pre-processing" Oct 30 22:25:36.255414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.255401 4335 cpu_manager.go:414] "containerMap: removing container" podUID="dd65776e-f29f-436a-aabb-e9e6b6d3497a" containerName="payload-processing" Oct 30 22:25:36.255414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.255405 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd65776e-f29f-436a-aabb-e9e6b6d3497a" containerName="payload-processing" Oct 30 22:25:36.255577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.255421 4335 cpu_manager.go:414] "containerMap: removing container" podUID="472bc81d-c1a3-463d-98fe-b98d4576aab7" containerName="payload-pre-processing" Oct 30 22:25:36.255577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.255426 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472bc81d-c1a3-463d-98fe-b98d4576aab7" containerName="payload-pre-processing" Oct 30 22:25:36.255577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.255509 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fbff4d0-50fb-48f1-bfe6-fa97318fa8a1" containerName="payload-pre-processing" Oct 30 22:25:36.255577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.255519 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd65776e-f29f-436a-aabb-e9e6b6d3497a" containerName="payload-processing" Oct 30 22:25:36.255577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.255527 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="472bc81d-c1a3-463d-98fe-b98d4576aab7" containerName="payload-pre-processing" Oct 30 22:25:36.255577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.255538 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1388034-f774-4af1-bdb7-fa9eb1ec07f5" containerName="payload-processing" Oct 30 22:25:36.260890 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.260872 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg" Oct 30 22:25:36.316978 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.316917 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg"] Oct 30 22:25:36.350570 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.350546 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg\" (UID: \"4f6c5ae4-4d9f-4efe-a696-a0780affb25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg" Oct 30 22:25:36.350747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.350591 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg\" (UID: \"4f6c5ae4-4d9f-4efe-a696-a0780affb25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg" Oct 30 22:25:36.350814 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.350794 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m862c\" (UniqueName: \"kubernetes.io/projected/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-kube-api-access-m862c\") pod \"maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg\" (UID: \"4f6c5ae4-4d9f-4efe-a696-a0780affb25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg" Oct 30 22:25:36.360502 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.360481 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:25:36.360651 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.360542 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:25:36.360651 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.360564 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:25:36.362349 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.362331 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:25:36.451704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.451675 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg\" (UID: \"4f6c5ae4-4d9f-4efe-a696-a0780affb25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg" Oct 30 22:25:36.451817 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.451718 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg\" (UID: \"4f6c5ae4-4d9f-4efe-a696-a0780affb25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg" Oct 30 22:25:36.451817 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.451784 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m862c\" (UniqueName: \"kubernetes.io/projected/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-kube-api-access-m862c\") pod \"maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg\" (UID: \"4f6c5ae4-4d9f-4efe-a696-a0780affb25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg" Oct 30 22:25:36.452427 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.452410 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg\" (UID: \"4f6c5ae4-4d9f-4efe-a696-a0780affb25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg" Oct 30 22:25:36.454268 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.454247 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg\" (UID: \"4f6c5ae4-4d9f-4efe-a696-a0780affb25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg" Oct 30 22:25:36.512576 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.512521 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m862c\" (UniqueName: \"kubernetes.io/projected/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-kube-api-access-m862c\") pod \"maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg\" (UID: \"4f6c5ae4-4d9f-4efe-a696-a0780affb25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg" Oct 30 22:25:36.575067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.575044 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg" Oct 30 22:25:36.954854 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:36.954828 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg"] Oct 30 22:25:36.955013 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:25:36.954950 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6c5ae4_4d9f_4efe_a696_a0780affb25d.slice/crio-c2ac9aa6913299774359b83289c2aad5a708ac51ab16893af8e0ce60f4036a30 WatchSource:0}: Error finding container c2ac9aa6913299774359b83289c2aad5a708ac51ab16893af8e0ce60f4036a30: Status 404 returned error can't find the container with id c2ac9aa6913299774359b83289c2aad5a708ac51ab16893af8e0ce60f4036a30 Oct 30 22:25:37.307143 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:37.307113 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg" event={"ID":"4f6c5ae4-4d9f-4efe-a696-a0780affb25d","Type":"ContainerStarted","Data":"c2ac9aa6913299774359b83289c2aad5a708ac51ab16893af8e0ce60f4036a30"} Oct 30 22:25:38.316574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:38.316542 4335 generic.go:356] "Generic (PLEG): container finished" podID="4f6c5ae4-4d9f-4efe-a696-a0780affb25d" containerID="923e9e70cabd94e43191170017afa399fa2b14cefa3df3689386f24466c25558" exitCode=0 Oct 30 22:25:38.317004 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:38.316658 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg" event={"ID":"4f6c5ae4-4d9f-4efe-a696-a0780affb25d","Type":"ContainerDied","Data":"923e9e70cabd94e43191170017afa399fa2b14cefa3df3689386f24466c25558"} Oct 30 22:25:38.485295 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:38.485251 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh"] Oct 30 22:25:38.485662 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:38.485590 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" podUID="fc259675-c041-4deb-aaf3-ac3c90c48a05" containerName="istio-proxy" containerID="cri-o://ac2da5b43441976bfaba61c971f747b569c528b5488e657d94b37a75f4d27e84" gracePeriod=30 Oct 30 22:25:39.475556 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:39.475530 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg" Oct 30 22:25:39.483944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:39.483922 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-maas-api-openshift-ca\") pod \"4f6c5ae4-4d9f-4efe-a696-a0780affb25d\" (UID: \"4f6c5ae4-4d9f-4efe-a696-a0780affb25d\") " Oct 30 22:25:39.484070 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:39.483998 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-maas-api-xks-ca\") pod \"4f6c5ae4-4d9f-4efe-a696-a0780affb25d\" (UID: \"4f6c5ae4-4d9f-4efe-a696-a0780affb25d\") " Oct 30 22:25:39.484133 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:39.484117 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-kube-api-access-m862c\" (UniqueName: \"kubernetes.io/projected/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-kube-api-access-m862c\") pod \"4f6c5ae4-4d9f-4efe-a696-a0780affb25d\" (UID: \"4f6c5ae4-4d9f-4efe-a696-a0780affb25d\") " Oct 30 22:25:39.484302 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:39.484274 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-maas-api-openshift-ca" pod "4f6c5ae4-4d9f-4efe-a696-a0780affb25d" (UID: "4f6c5ae4-4d9f-4efe-a696-a0780affb25d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:39.484709 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:39.484633 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:39.486136 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:39.486115 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-maas-api-xks-ca" pod "4f6c5ae4-4d9f-4efe-a696-a0780affb25d" (UID: "4f6c5ae4-4d9f-4efe-a696-a0780affb25d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:39.486212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:39.486185 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-kube-api-access-m862c" pod "4f6c5ae4-4d9f-4efe-a696-a0780affb25d" (UID: "4f6c5ae4-4d9f-4efe-a696-a0780affb25d"). InnerVolumeSpecName "kube-api-access-m862c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:39.586295 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:39.586236 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:39.586295 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:39.586264 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m862c\" (UniqueName: \"kubernetes.io/projected/4f6c5ae4-4d9f-4efe-a696-a0780affb25d-kube-api-access-m862c\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:40.334060 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:40.334030 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg" Oct 30 22:25:40.334227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:40.334068 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg" event={"ID":"4f6c5ae4-4d9f-4efe-a696-a0780affb25d","Type":"ContainerDied","Data":"c2ac9aa6913299774359b83289c2aad5a708ac51ab16893af8e0ce60f4036a30"} Oct 30 22:25:40.334227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:40.334117 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2ac9aa6913299774359b83289c2aad5a708ac51ab16893af8e0ce60f4036a30" Oct 30 22:25:40.746385 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:40.746304 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk"] Oct 30 22:25:40.761769 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:40.761729 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr"] Oct 30 22:25:41.924898 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:41.924860 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf"] Oct 30 22:25:41.925466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:41.925452 4335 cpu_manager.go:414] "containerMap: removing container" podUID="4f6c5ae4-4d9f-4efe-a696-a0780affb25d" containerName="revoke-keys" Oct 30 22:25:41.925518 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:41.925468 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f6c5ae4-4d9f-4efe-a696-a0780affb25d" containerName="revoke-keys" Oct 30 22:25:41.925609 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:41.925599 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f6c5ae4-4d9f-4efe-a696-a0780affb25d" containerName="revoke-keys" Oct 30 22:25:41.929685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:41.929665 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf" Oct 30 22:25:41.987684 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:41.987650 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf"] Oct 30 22:25:42.018182 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:42.018150 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be962e0f-ae2d-4805-964e-fff48077329e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf\" (UID: \"be962e0f-ae2d-4805-964e-fff48077329e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf" Oct 30 22:25:42.018366 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:42.018312 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm2t4\" (UniqueName: \"kubernetes.io/projected/be962e0f-ae2d-4805-964e-fff48077329e-kube-api-access-dm2t4\") pod \"maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf\" (UID: \"be962e0f-ae2d-4805-964e-fff48077329e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf" Oct 30 22:25:42.018366 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:42.018353 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be962e0f-ae2d-4805-964e-fff48077329e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf\" (UID: \"be962e0f-ae2d-4805-964e-fff48077329e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf" Oct 30 22:25:42.030092 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:42.030069 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:25:42.030227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:42.030069 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:25:42.030227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:42.030140 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:25:42.031138 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:42.031121 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:25:42.119136 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:42.119110 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be962e0f-ae2d-4805-964e-fff48077329e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf\" (UID: \"be962e0f-ae2d-4805-964e-fff48077329e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf" Oct 30 22:25:42.119259 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:42.119206 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm2t4\" (UniqueName: \"kubernetes.io/projected/be962e0f-ae2d-4805-964e-fff48077329e-kube-api-access-dm2t4\") pod \"maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf\" (UID: \"be962e0f-ae2d-4805-964e-fff48077329e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf" Oct 30 22:25:42.119259 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:42.119230 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be962e0f-ae2d-4805-964e-fff48077329e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf\" (UID: \"be962e0f-ae2d-4805-964e-fff48077329e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf" Oct 30 22:25:42.119863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:42.119845 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be962e0f-ae2d-4805-964e-fff48077329e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf\" (UID: \"be962e0f-ae2d-4805-964e-fff48077329e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf" Oct 30 22:25:42.121583 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:42.121566 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be962e0f-ae2d-4805-964e-fff48077329e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf\" (UID: \"be962e0f-ae2d-4805-964e-fff48077329e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf" Oct 30 22:25:42.180324 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:42.180264 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm2t4\" (UniqueName: \"kubernetes.io/projected/be962e0f-ae2d-4805-964e-fff48077329e-kube-api-access-dm2t4\") pod \"maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf\" (UID: \"be962e0f-ae2d-4805-964e-fff48077329e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf" Oct 30 22:25:42.244068 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:42.244044 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf" Oct 30 22:25:42.627559 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:25:42.627520 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe962e0f_ae2d_4805_964e_fff48077329e.slice/crio-06da848a62f6cb0197cff730bdda20b94e896a25e98082877ca75baa91f12bdd WatchSource:0}: Error finding container 06da848a62f6cb0197cff730bdda20b94e896a25e98082877ca75baa91f12bdd: Status 404 returned error can't find the container with id 06da848a62f6cb0197cff730bdda20b94e896a25e98082877ca75baa91f12bdd Oct 30 22:25:42.628705 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:42.628502 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf"] Oct 30 22:25:43.360539 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.360510 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf" event={"ID":"be962e0f-ae2d-4805-964e-fff48077329e","Type":"ContainerStarted","Data":"6aa4874772db37fa397a475fb3343660f562623fed8de034b96495379a6bf7d2"} Oct 30 22:25:43.360960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.360561 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf" event={"ID":"be962e0f-ae2d-4805-964e-fff48077329e","Type":"ContainerStarted","Data":"06da848a62f6cb0197cff730bdda20b94e896a25e98082877ca75baa91f12bdd"} Oct 30 22:25:43.477384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.477336 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf" podStartSLOduration=2.477322484 podStartE2EDuration="2.477322484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:43.474897226 +0000 UTC m=+2476.575328959" watchObservedRunningTime="2026-10-30 22:25:43.477322484 +0000 UTC m=+2476.577754215" Oct 30 22:25:43.790494 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.790463 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:25:43.841056 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.841028 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc259675-c041-4deb-aaf3-ac3c90c48a05-kube-api-access-zssjx\" (UniqueName: \"kubernetes.io/projected/fc259675-c041-4deb-aaf3-ac3c90c48a05-kube-api-access-zssjx\") pod \"fc259675-c041-4deb-aaf3-ac3c90c48a05\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " Oct 30 22:25:43.841218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.841082 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-workload-socket\") pod \"fc259675-c041-4deb-aaf3-ac3c90c48a05\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " Oct 30 22:25:43.841218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.841114 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-podinfo\") pod \"fc259675-c041-4deb-aaf3-ac3c90c48a05\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " Oct 30 22:25:43.841218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.841164 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-credential-socket\") pod \"fc259675-c041-4deb-aaf3-ac3c90c48a05\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " Oct 30 22:25:43.841218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.841218 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-workload-certs\") pod \"fc259675-c041-4deb-aaf3-ac3c90c48a05\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " Oct 30 22:25:43.841466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.841284 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-data\") pod \"fc259675-c041-4deb-aaf3-ac3c90c48a05\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " Oct 30 22:25:43.841466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.841307 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-token\" (UniqueName: \"kubernetes.io/projected/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-token\") pod \"fc259675-c041-4deb-aaf3-ac3c90c48a05\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " Oct 30 22:25:43.841466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.841344 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-envoy\") pod \"fc259675-c041-4deb-aaf3-ac3c90c48a05\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " Oct 30 22:25:43.841466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.841447 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc259675-c041-4deb-aaf3-ac3c90c48a05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc259675-c041-4deb-aaf3-ac3c90c48a05-istiod-ca-cert\") pod \"fc259675-c041-4deb-aaf3-ac3c90c48a05\" (UID: \"fc259675-c041-4deb-aaf3-ac3c90c48a05\") " Oct 30 22:25:43.841747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.841477 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-workload-socket" pod "fc259675-c041-4deb-aaf3-ac3c90c48a05" (UID: "fc259675-c041-4deb-aaf3-ac3c90c48a05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.841747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.841569 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-credential-socket" pod "fc259675-c041-4deb-aaf3-ac3c90c48a05" (UID: "fc259675-c041-4deb-aaf3-ac3c90c48a05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.842034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.841949 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-workload-certs" pod "fc259675-c041-4deb-aaf3-ac3c90c48a05" (UID: "fc259675-c041-4deb-aaf3-ac3c90c48a05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.842034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.841980 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc259675-c041-4deb-aaf3-ac3c90c48a05-istiod-ca-cert" pod "fc259675-c041-4deb-aaf3-ac3c90c48a05" (UID: "fc259675-c041-4deb-aaf3-ac3c90c48a05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:43.842034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.842004 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-data" pod "fc259675-c041-4deb-aaf3-ac3c90c48a05" (UID: "fc259675-c041-4deb-aaf3-ac3c90c48a05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.842200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.842088 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc259675-c041-4deb-aaf3-ac3c90c48a05-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.842200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.842106 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.842200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.842118 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.842200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.842127 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.843743 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.843719 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc259675-c041-4deb-aaf3-ac3c90c48a05-kube-api-access-zssjx" pod "fc259675-c041-4deb-aaf3-ac3c90c48a05" (UID: "fc259675-c041-4deb-aaf3-ac3c90c48a05"). InnerVolumeSpecName "kube-api-access-zssjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:43.844152 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.844129 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-podinfo" pod "fc259675-c041-4deb-aaf3-ac3c90c48a05" (UID: "fc259675-c041-4deb-aaf3-ac3c90c48a05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:43.844249 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.844162 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-token" pod "fc259675-c041-4deb-aaf3-ac3c90c48a05" (UID: "fc259675-c041-4deb-aaf3-ac3c90c48a05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:43.844389 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.844373 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-envoy" pod "fc259675-c041-4deb-aaf3-ac3c90c48a05" (UID: "fc259675-c041-4deb-aaf3-ac3c90c48a05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:43.943716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.943511 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zssjx\" (UniqueName: \"kubernetes.io/projected/fc259675-c041-4deb-aaf3-ac3c90c48a05-kube-api-access-zssjx\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.943716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.943549 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.943716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.943564 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.943716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.943577 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:43.943716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:43.943591 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc259675-c041-4deb-aaf3-ac3c90c48a05-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:44.369486 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:44.369455 4335 generic.go:356] "Generic (PLEG): container finished" podID="be962e0f-ae2d-4805-964e-fff48077329e" containerID="6aa4874772db37fa397a475fb3343660f562623fed8de034b96495379a6bf7d2" exitCode=0 Oct 30 22:25:44.370022 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:44.369541 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf" event={"ID":"be962e0f-ae2d-4805-964e-fff48077329e","Type":"ContainerDied","Data":"6aa4874772db37fa397a475fb3343660f562623fed8de034b96495379a6bf7d2"} Oct 30 22:25:44.371424 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:44.371404 4335 generic.go:356] "Generic (PLEG): container finished" podID="fc259675-c041-4deb-aaf3-ac3c90c48a05" containerID="ac2da5b43441976bfaba61c971f747b569c528b5488e657d94b37a75f4d27e84" exitCode=0 Oct 30 22:25:44.371504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:44.371447 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" Oct 30 22:25:44.371504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:44.371483 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" event={"ID":"fc259675-c041-4deb-aaf3-ac3c90c48a05","Type":"ContainerDied","Data":"ac2da5b43441976bfaba61c971f747b569c528b5488e657d94b37a75f4d27e84"} Oct 30 22:25:44.371581 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:44.371539 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh" event={"ID":"fc259675-c041-4deb-aaf3-ac3c90c48a05","Type":"ContainerDied","Data":"a7793369bf0f616f7db9f008162ca71736a2c800b72d0cd75255b7a421b24b77"} Oct 30 22:25:44.371581 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:44.371569 4335 scope.go:122] "RemoveContainer" containerID="ac2da5b43441976bfaba61c971f747b569c528b5488e657d94b37a75f4d27e84" Oct 30 22:25:44.382923 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:44.382906 4335 scope.go:122] "RemoveContainer" containerID="6e9b65a2165157264b6ca29415de1e43815cc5a204a0cdcdcc1a7ba2f0ed62f8" Oct 30 22:25:44.393055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:44.393033 4335 scope.go:122] "RemoveContainer" containerID="ac2da5b43441976bfaba61c971f747b569c528b5488e657d94b37a75f4d27e84" Oct 30 22:25:44.393339 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:25:44.393320 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac2da5b43441976bfaba61c971f747b569c528b5488e657d94b37a75f4d27e84\": container with ID starting with ac2da5b43441976bfaba61c971f747b569c528b5488e657d94b37a75f4d27e84 not found: ID does not exist" containerID="ac2da5b43441976bfaba61c971f747b569c528b5488e657d94b37a75f4d27e84" Oct 30 22:25:44.393393 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:44.393351 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac2da5b43441976bfaba61c971f747b569c528b5488e657d94b37a75f4d27e84"} err="failed to get container status \"ac2da5b43441976bfaba61c971f747b569c528b5488e657d94b37a75f4d27e84\": rpc error: code = NotFound desc = could not find container \"ac2da5b43441976bfaba61c971f747b569c528b5488e657d94b37a75f4d27e84\": container with ID starting with ac2da5b43441976bfaba61c971f747b569c528b5488e657d94b37a75f4d27e84 not found: ID does not exist" Oct 30 22:25:44.393393 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:44.393371 4335 scope.go:122] "RemoveContainer" containerID="6e9b65a2165157264b6ca29415de1e43815cc5a204a0cdcdcc1a7ba2f0ed62f8" Oct 30 22:25:44.393628 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:25:44.393596 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e9b65a2165157264b6ca29415de1e43815cc5a204a0cdcdcc1a7ba2f0ed62f8\": container with ID starting with 6e9b65a2165157264b6ca29415de1e43815cc5a204a0cdcdcc1a7ba2f0ed62f8 not found: ID does not exist" containerID="6e9b65a2165157264b6ca29415de1e43815cc5a204a0cdcdcc1a7ba2f0ed62f8" Oct 30 22:25:44.393679 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:44.393637 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e9b65a2165157264b6ca29415de1e43815cc5a204a0cdcdcc1a7ba2f0ed62f8"} err="failed to get container status \"6e9b65a2165157264b6ca29415de1e43815cc5a204a0cdcdcc1a7ba2f0ed62f8\": rpc error: code = NotFound desc = could not find container \"6e9b65a2165157264b6ca29415de1e43815cc5a204a0cdcdcc1a7ba2f0ed62f8\": container with ID starting with 6e9b65a2165157264b6ca29415de1e43815cc5a204a0cdcdcc1a7ba2f0ed62f8 not found: ID does not exist" Oct 30 22:25:44.600508 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:44.600478 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh"] Oct 30 22:25:44.605975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:44.605952 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0f3344-openshift-default-94647df7c-nxqqh"] Oct 30 22:25:45.058724 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:45.058691 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc259675-c041-4deb-aaf3-ac3c90c48a05" path="/var/lib/kubelet/pods/fc259675-c041-4deb-aaf3-ac3c90c48a05/volumes" Oct 30 22:25:45.529425 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:45.529395 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf" Oct 30 22:25:45.663276 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:45.663194 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be962e0f-ae2d-4805-964e-fff48077329e-kube-api-access-dm2t4\" (UniqueName: \"kubernetes.io/projected/be962e0f-ae2d-4805-964e-fff48077329e-kube-api-access-dm2t4\") pod \"be962e0f-ae2d-4805-964e-fff48077329e\" (UID: \"be962e0f-ae2d-4805-964e-fff48077329e\") " Oct 30 22:25:45.663276 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:45.663274 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be962e0f-ae2d-4805-964e-fff48077329e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be962e0f-ae2d-4805-964e-fff48077329e-maas-api-openshift-ca\") pod \"be962e0f-ae2d-4805-964e-fff48077329e\" (UID: \"be962e0f-ae2d-4805-964e-fff48077329e\") " Oct 30 22:25:45.663507 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:45.663374 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be962e0f-ae2d-4805-964e-fff48077329e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be962e0f-ae2d-4805-964e-fff48077329e-maas-api-xks-ca\") pod \"be962e0f-ae2d-4805-964e-fff48077329e\" (UID: \"be962e0f-ae2d-4805-964e-fff48077329e\") " Oct 30 22:25:45.663708 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:45.663680 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be962e0f-ae2d-4805-964e-fff48077329e-maas-api-openshift-ca" pod "be962e0f-ae2d-4805-964e-fff48077329e" (UID: "be962e0f-ae2d-4805-964e-fff48077329e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:45.663926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:45.663904 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be962e0f-ae2d-4805-964e-fff48077329e-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:45.665590 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:45.665562 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be962e0f-ae2d-4805-964e-fff48077329e-maas-api-xks-ca" pod "be962e0f-ae2d-4805-964e-fff48077329e" (UID: "be962e0f-ae2d-4805-964e-fff48077329e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:45.666142 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:45.666119 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be962e0f-ae2d-4805-964e-fff48077329e-kube-api-access-dm2t4" pod "be962e0f-ae2d-4805-964e-fff48077329e" (UID: "be962e0f-ae2d-4805-964e-fff48077329e"). InnerVolumeSpecName "kube-api-access-dm2t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:45.765276 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:45.765246 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be962e0f-ae2d-4805-964e-fff48077329e-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:45.765276 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:45.765273 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm2t4\" (UniqueName: \"kubernetes.io/projected/be962e0f-ae2d-4805-964e-fff48077329e-kube-api-access-dm2t4\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:46.365875 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:46.365845 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf"] Oct 30 22:25:46.391191 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:46.391166 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf" Oct 30 22:25:46.391344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:46.391170 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf" event={"ID":"be962e0f-ae2d-4805-964e-fff48077329e","Type":"ContainerDied","Data":"06da848a62f6cb0197cff730bdda20b94e896a25e98082877ca75baa91f12bdd"} Oct 30 22:25:46.391344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:46.391275 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06da848a62f6cb0197cff730bdda20b94e896a25e98082877ca75baa91f12bdd" Oct 30 22:25:46.415800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:46.415766 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd"] Oct 30 22:25:46.641894 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:46.641814 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq"] Oct 30 22:25:46.732309 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:46.732284 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl"] Oct 30 22:25:47.705697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.705666 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p"] Oct 30 22:25:47.706323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.706301 4335 cpu_manager.go:414] "containerMap: removing container" podUID="be962e0f-ae2d-4805-964e-fff48077329e" containerName="revoke-keys" Oct 30 22:25:47.706323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.706318 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be962e0f-ae2d-4805-964e-fff48077329e" containerName="revoke-keys" Oct 30 22:25:47.706323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.706327 4335 cpu_manager.go:414] "containerMap: removing container" podUID="fc259675-c041-4deb-aaf3-ac3c90c48a05" containerName="istio-proxy" Oct 30 22:25:47.706575 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.706333 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc259675-c041-4deb-aaf3-ac3c90c48a05" containerName="istio-proxy" Oct 30 22:25:47.706575 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.706446 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="be962e0f-ae2d-4805-964e-fff48077329e" containerName="revoke-keys" Oct 30 22:25:47.706575 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.706453 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc259675-c041-4deb-aaf3-ac3c90c48a05" containerName="istio-proxy" Oct 30 22:25:47.706575 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.706464 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc259675-c041-4deb-aaf3-ac3c90c48a05" containerName="istio-proxy" Oct 30 22:25:47.710090 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.710072 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.791488 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.791453 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-credential-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.791708 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.791499 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istiod-ca-cert\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.791708 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.791667 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-podinfo\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.791708 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.791701 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-data\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.791906 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.791860 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-workload-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.791967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.791958 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-envoy\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.792028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.791988 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-token\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.792093 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.792021 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-workload-certs\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.792192 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.792166 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2ss6\" (UniqueName: \"kubernetes.io/projected/60a004f7-2dc7-4b0e-a844-48404c0e35f2-kube-api-access-t2ss6\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.806430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.806401 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p"] Oct 30 22:25:47.893856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.893827 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-workload-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.894017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.893907 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-envoy\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.894017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.893927 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-token\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.894017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.893947 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-workload-certs\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.894017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.893987 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2ss6\" (UniqueName: \"kubernetes.io/projected/60a004f7-2dc7-4b0e-a844-48404c0e35f2-kube-api-access-t2ss6\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.894213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.894111 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-credential-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.894213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.894160 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istiod-ca-cert\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.894339 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.894249 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-workload-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.894339 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.894270 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-podinfo\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.894339 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.894302 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-data\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.894498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.894379 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-workload-certs\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.894498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.894446 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-credential-socket\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.894679 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.894663 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-data\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.894893 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.894870 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istiod-ca-cert\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.896472 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.896449 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-envoy\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.896910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.896889 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-podinfo\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.955155 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.955126 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-token\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:47.955534 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:47.955507 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2ss6\" (UniqueName: \"kubernetes.io/projected/60a004f7-2dc7-4b0e-a844-48404c0e35f2-kube-api-access-t2ss6\") pod \"e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:48.024436 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:48.024374 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:48.470072 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:48.467694 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p"] Oct 30 22:25:48.479444 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:48.479382 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:25:48.479563 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:48.479478 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:25:48.479563 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:48.479514 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:25:49.444090 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:49.444045 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" event={"ID":"60a004f7-2dc7-4b0e-a844-48404c0e35f2","Type":"ContainerStarted","Data":"e8df66045448c86fea758dc8bdf2cfedf466d46e4c4df435f9b958c926602c1c"} Oct 30 22:25:49.444604 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:49.444123 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" event={"ID":"60a004f7-2dc7-4b0e-a844-48404c0e35f2","Type":"ContainerStarted","Data":"64d89ebbefbf26bb607fea4029024203d669c5343ddf0d331abfa821c3f40409"} Oct 30 22:25:50.027414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:50.027383 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:50.034505 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:50.034471 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 30 22:25:50.034686 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:50.034548 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" podUID="60a004f7-2dc7-4b0e-a844-48404c0e35f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 30 22:25:51.027008 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:51.026973 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 30 22:25:51.027539 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:51.027037 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" podUID="60a004f7-2dc7-4b0e-a844-48404c0e35f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 30 22:25:51.642287 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:51.642240 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq" podUID="b3ca0831-b8aa-487f-8745-3af2e1d3270e" containerName="payload-processing" containerID="cri-o://a53179c91698c9b7b03e59aca76cdb7cb7c389a6e356de8e8238a49d67904788" gracePeriod=25 Oct 30 22:25:51.732895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:51.732852 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl" podUID="87dc45da-3324-4fc9-bda4-7186b19d9a64" containerName="payload-pre-processing" containerID="cri-o://9b26e01f073682aab5fed4f5917a247dd4e0c025622c6beaa2d900b1fbab19ee" gracePeriod=25 Oct 30 22:25:52.041213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.039774 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 30 22:25:52.041213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.041051 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" podUID="60a004f7-2dc7-4b0e-a844-48404c0e35f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 30 22:25:52.075292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.075269 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq" Oct 30 22:25:52.157128 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.157105 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl" Oct 30 22:25:52.170088 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.170067 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ca0831-b8aa-487f-8745-3af2e1d3270e-kube-api-access-s6769\" (UniqueName: \"kubernetes.io/projected/b3ca0831-b8aa-487f-8745-3af2e1d3270e-kube-api-access-s6769\") pod \"b3ca0831-b8aa-487f-8745-3af2e1d3270e\" (UID: \"b3ca0831-b8aa-487f-8745-3af2e1d3270e\") " Oct 30 22:25:52.170192 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.170115 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ca0831-b8aa-487f-8745-3af2e1d3270e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ca0831-b8aa-487f-8745-3af2e1d3270e-plugins-config-volume\") pod \"b3ca0831-b8aa-487f-8745-3af2e1d3270e\" (UID: \"b3ca0831-b8aa-487f-8745-3af2e1d3270e\") " Oct 30 22:25:52.170540 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.170514 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ca0831-b8aa-487f-8745-3af2e1d3270e-plugins-config-volume" pod "b3ca0831-b8aa-487f-8745-3af2e1d3270e" (UID: "b3ca0831-b8aa-487f-8745-3af2e1d3270e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:52.172212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.172190 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ca0831-b8aa-487f-8745-3af2e1d3270e-kube-api-access-s6769" pod "b3ca0831-b8aa-487f-8745-3af2e1d3270e" (UID: "b3ca0831-b8aa-487f-8745-3af2e1d3270e"). InnerVolumeSpecName "kube-api-access-s6769". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:52.195445 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.195371 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" podStartSLOduration=5.195359295 podStartE2EDuration="5.195359295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:49.626958916 +0000 UTC m=+2482.727390650" watchObservedRunningTime="2026-10-30 22:25:52.195359295 +0000 UTC m=+2485.295791027" Oct 30 22:25:52.271235 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.271204 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87dc45da-3324-4fc9-bda4-7186b19d9a64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87dc45da-3324-4fc9-bda4-7186b19d9a64-plugins-config-volume\") pod \"87dc45da-3324-4fc9-bda4-7186b19d9a64\" (UID: \"87dc45da-3324-4fc9-bda4-7186b19d9a64\") " Oct 30 22:25:52.271455 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.271321 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87dc45da-3324-4fc9-bda4-7186b19d9a64-kube-api-access-bpv74\" (UniqueName: \"kubernetes.io/projected/87dc45da-3324-4fc9-bda4-7186b19d9a64-kube-api-access-bpv74\") pod \"87dc45da-3324-4fc9-bda4-7186b19d9a64\" (UID: \"87dc45da-3324-4fc9-bda4-7186b19d9a64\") " Oct 30 22:25:52.271689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.271656 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87dc45da-3324-4fc9-bda4-7186b19d9a64-plugins-config-volume" pod "87dc45da-3324-4fc9-bda4-7186b19d9a64" (UID: "87dc45da-3324-4fc9-bda4-7186b19d9a64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:52.271842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.271823 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87dc45da-3324-4fc9-bda4-7186b19d9a64-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:52.271895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.271849 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6769\" (UniqueName: \"kubernetes.io/projected/b3ca0831-b8aa-487f-8745-3af2e1d3270e-kube-api-access-s6769\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:52.271895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.271866 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ca0831-b8aa-487f-8745-3af2e1d3270e-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:52.273548 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.273526 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87dc45da-3324-4fc9-bda4-7186b19d9a64-kube-api-access-bpv74" pod "87dc45da-3324-4fc9-bda4-7186b19d9a64" (UID: "87dc45da-3324-4fc9-bda4-7186b19d9a64"). InnerVolumeSpecName "kube-api-access-bpv74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:52.372747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.372714 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpv74\" (UniqueName: \"kubernetes.io/projected/87dc45da-3324-4fc9-bda4-7186b19d9a64-kube-api-access-bpv74\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:52.436998 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.436963 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh"] Oct 30 22:25:52.477092 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.477012 4335 generic.go:356] "Generic (PLEG): container finished" podID="b3ca0831-b8aa-487f-8745-3af2e1d3270e" containerID="a53179c91698c9b7b03e59aca76cdb7cb7c389a6e356de8e8238a49d67904788" exitCode=0 Oct 30 22:25:52.477092 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.477058 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq" Oct 30 22:25:52.477301 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.477089 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq" event={"ID":"b3ca0831-b8aa-487f-8745-3af2e1d3270e","Type":"ContainerDied","Data":"a53179c91698c9b7b03e59aca76cdb7cb7c389a6e356de8e8238a49d67904788"} Oct 30 22:25:52.477301 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.477153 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq" event={"ID":"b3ca0831-b8aa-487f-8745-3af2e1d3270e","Type":"ContainerDied","Data":"1350007bc532daa33ba8e3aeca6095ff4bac707385c5aeb77bb0e39747a349cc"} Oct 30 22:25:52.477301 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.477176 4335 scope.go:122] "RemoveContainer" containerID="a53179c91698c9b7b03e59aca76cdb7cb7c389a6e356de8e8238a49d67904788" Oct 30 22:25:52.478726 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.478705 4335 generic.go:356] "Generic (PLEG): container finished" podID="87dc45da-3324-4fc9-bda4-7186b19d9a64" containerID="9b26e01f073682aab5fed4f5917a247dd4e0c025622c6beaa2d900b1fbab19ee" exitCode=0 Oct 30 22:25:52.478828 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.478746 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl" Oct 30 22:25:52.478828 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.478780 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl" event={"ID":"87dc45da-3324-4fc9-bda4-7186b19d9a64","Type":"ContainerDied","Data":"9b26e01f073682aab5fed4f5917a247dd4e0c025622c6beaa2d900b1fbab19ee"} Oct 30 22:25:52.478828 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.478818 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl" event={"ID":"87dc45da-3324-4fc9-bda4-7186b19d9a64","Type":"ContainerDied","Data":"7a3d9f704b415be70322d42712a0a026b3d73c94654897bb1c6b12b184e726f6"} Oct 30 22:25:52.491236 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.491215 4335 scope.go:122] "RemoveContainer" containerID="a53179c91698c9b7b03e59aca76cdb7cb7c389a6e356de8e8238a49d67904788" Oct 30 22:25:52.491516 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:25:52.491497 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a53179c91698c9b7b03e59aca76cdb7cb7c389a6e356de8e8238a49d67904788\": container with ID starting with a53179c91698c9b7b03e59aca76cdb7cb7c389a6e356de8e8238a49d67904788 not found: ID does not exist" containerID="a53179c91698c9b7b03e59aca76cdb7cb7c389a6e356de8e8238a49d67904788" Oct 30 22:25:52.491565 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.491524 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53179c91698c9b7b03e59aca76cdb7cb7c389a6e356de8e8238a49d67904788"} err="failed to get container status \"a53179c91698c9b7b03e59aca76cdb7cb7c389a6e356de8e8238a49d67904788\": rpc error: code = NotFound desc = could not find container \"a53179c91698c9b7b03e59aca76cdb7cb7c389a6e356de8e8238a49d67904788\": container with ID starting with a53179c91698c9b7b03e59aca76cdb7cb7c389a6e356de8e8238a49d67904788 not found: ID does not exist" Oct 30 22:25:52.491565 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.491539 4335 scope.go:122] "RemoveContainer" containerID="9b26e01f073682aab5fed4f5917a247dd4e0c025622c6beaa2d900b1fbab19ee" Oct 30 22:25:52.535889 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.535868 4335 scope.go:122] "RemoveContainer" containerID="9b26e01f073682aab5fed4f5917a247dd4e0c025622c6beaa2d900b1fbab19ee" Oct 30 22:25:52.536236 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:25:52.536210 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b26e01f073682aab5fed4f5917a247dd4e0c025622c6beaa2d900b1fbab19ee\": container with ID starting with 9b26e01f073682aab5fed4f5917a247dd4e0c025622c6beaa2d900b1fbab19ee not found: ID does not exist" containerID="9b26e01f073682aab5fed4f5917a247dd4e0c025622c6beaa2d900b1fbab19ee" Oct 30 22:25:52.536321 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.536249 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b26e01f073682aab5fed4f5917a247dd4e0c025622c6beaa2d900b1fbab19ee"} err="failed to get container status \"9b26e01f073682aab5fed4f5917a247dd4e0c025622c6beaa2d900b1fbab19ee\": rpc error: code = NotFound desc = could not find container \"9b26e01f073682aab5fed4f5917a247dd4e0c025622c6beaa2d900b1fbab19ee\": container with ID starting with 9b26e01f073682aab5fed4f5917a247dd4e0c025622c6beaa2d900b1fbab19ee not found: ID does not exist" Oct 30 22:25:52.684243 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.684207 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl"] Oct 30 22:25:52.690459 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.690427 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-cef074-85d6b8c7f9-wgrvl"] Oct 30 22:25:52.847748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.845834 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq"] Oct 30 22:25:52.854038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:52.854009 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cef074-5777bd7b7f-fv4sq"] Oct 30 22:25:53.031977 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:53.031934 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 30 22:25:53.032212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:53.032000 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" podUID="60a004f7-2dc7-4b0e-a844-48404c0e35f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 30 22:25:53.059015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:53.058985 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87dc45da-3324-4fc9-bda4-7186b19d9a64" path="/var/lib/kubelet/pods/87dc45da-3324-4fc9-bda4-7186b19d9a64/volumes" Oct 30 22:25:53.059375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:53.059361 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ca0831-b8aa-487f-8745-3af2e1d3270e" path="/var/lib/kubelet/pods/b3ca0831-b8aa-487f-8745-3af2e1d3270e/volumes" Oct 30 22:25:54.028639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.027909 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 30 22:25:54.028639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.027976 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" podUID="60a004f7-2dc7-4b0e-a844-48404c0e35f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 30 22:25:54.872903 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.872875 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb"] Oct 30 22:25:54.873480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.873466 4335 cpu_manager.go:414] "containerMap: removing container" podUID="87dc45da-3324-4fc9-bda4-7186b19d9a64" containerName="payload-pre-processing" Oct 30 22:25:54.873521 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.873481 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87dc45da-3324-4fc9-bda4-7186b19d9a64" containerName="payload-pre-processing" Oct 30 22:25:54.873521 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.873499 4335 cpu_manager.go:414] "containerMap: removing container" podUID="fc259675-c041-4deb-aaf3-ac3c90c48a05" containerName="istio-proxy" Oct 30 22:25:54.873521 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.873504 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc259675-c041-4deb-aaf3-ac3c90c48a05" containerName="istio-proxy" Oct 30 22:25:54.873636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.873526 4335 cpu_manager.go:414] "containerMap: removing container" podUID="b3ca0831-b8aa-487f-8745-3af2e1d3270e" containerName="payload-processing" Oct 30 22:25:54.873636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.873530 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ca0831-b8aa-487f-8745-3af2e1d3270e" containerName="payload-processing" Oct 30 22:25:54.873636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.873632 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="87dc45da-3324-4fc9-bda4-7186b19d9a64" containerName="payload-pre-processing" Oct 30 22:25:54.873743 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.873645 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ca0831-b8aa-487f-8745-3af2e1d3270e" containerName="payload-processing" Oct 30 22:25:54.877237 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.877222 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb" Oct 30 22:25:54.932318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.932290 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9"] Oct 30 22:25:54.937413 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.937387 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9" Oct 30 22:25:54.937762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.937741 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb"] Oct 30 22:25:54.949153 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.949130 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d854e6-f96e-4c8f-906a-9f2a46eabc35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d14dca27-hmbtb\" (UID: \"97d854e6-f96e-4c8f-906a-9f2a46eabc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb" Oct 30 22:25:54.949287 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.949179 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d854e6-f96e-4c8f-906a-9f2a46eabc35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d14dca27-hmbtb\" (UID: \"97d854e6-f96e-4c8f-906a-9f2a46eabc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb" Oct 30 22:25:54.949287 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.949252 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjkds\" (UniqueName: \"kubernetes.io/projected/97d854e6-f96e-4c8f-906a-9f2a46eabc35-kube-api-access-sjkds\") pod \"maas-api-revoke-sub-d14dca27-hmbtb\" (UID: \"97d854e6-f96e-4c8f-906a-9f2a46eabc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb" Oct 30 22:25:54.974955 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.974923 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:25:54.975054 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.974953 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:25:54.981474 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.981452 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:25:54.981474 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:54.981469 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:25:55.011159 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.011123 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9"] Oct 30 22:25:55.024921 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.024891 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 30 22:25:55.025039 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.024940 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" podUID="60a004f7-2dc7-4b0e-a844-48404c0e35f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 30 22:25:55.051106 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.051076 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjkds\" (UniqueName: \"kubernetes.io/projected/97d854e6-f96e-4c8f-906a-9f2a46eabc35-kube-api-access-sjkds\") pod \"maas-api-revoke-sub-d14dca27-hmbtb\" (UID: \"97d854e6-f96e-4c8f-906a-9f2a46eabc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb" Oct 30 22:25:55.051240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.051190 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffddb\" (UniqueName: \"kubernetes.io/projected/b1655cc1-72e4-4f89-a69f-b351d3228a1c-kube-api-access-ffddb\") pod \"maas-api-revoke-sub-81565b10-gwgm9\" (UID: \"b1655cc1-72e4-4f89-a69f-b351d3228a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9" Oct 30 22:25:55.051304 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.051270 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d854e6-f96e-4c8f-906a-9f2a46eabc35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d14dca27-hmbtb\" (UID: \"97d854e6-f96e-4c8f-906a-9f2a46eabc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb" Oct 30 22:25:55.051388 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.051356 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d854e6-f96e-4c8f-906a-9f2a46eabc35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d14dca27-hmbtb\" (UID: \"97d854e6-f96e-4c8f-906a-9f2a46eabc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb" Oct 30 22:25:55.051505 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.051408 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1655cc1-72e4-4f89-a69f-b351d3228a1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81565b10-gwgm9\" (UID: \"b1655cc1-72e4-4f89-a69f-b351d3228a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9" Oct 30 22:25:55.051505 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.051470 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1655cc1-72e4-4f89-a69f-b351d3228a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81565b10-gwgm9\" (UID: \"b1655cc1-72e4-4f89-a69f-b351d3228a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9" Oct 30 22:25:55.051995 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.051974 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d854e6-f96e-4c8f-906a-9f2a46eabc35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d14dca27-hmbtb\" (UID: \"97d854e6-f96e-4c8f-906a-9f2a46eabc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb" Oct 30 22:25:55.053932 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.053909 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d854e6-f96e-4c8f-906a-9f2a46eabc35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d14dca27-hmbtb\" (UID: \"97d854e6-f96e-4c8f-906a-9f2a46eabc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb" Oct 30 22:25:55.112981 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.112946 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjkds\" (UniqueName: \"kubernetes.io/projected/97d854e6-f96e-4c8f-906a-9f2a46eabc35-kube-api-access-sjkds\") pod \"maas-api-revoke-sub-d14dca27-hmbtb\" (UID: \"97d854e6-f96e-4c8f-906a-9f2a46eabc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb" Oct 30 22:25:55.152671 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.152579 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffddb\" (UniqueName: \"kubernetes.io/projected/b1655cc1-72e4-4f89-a69f-b351d3228a1c-kube-api-access-ffddb\") pod \"maas-api-revoke-sub-81565b10-gwgm9\" (UID: \"b1655cc1-72e4-4f89-a69f-b351d3228a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9" Oct 30 22:25:55.152828 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.152696 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1655cc1-72e4-4f89-a69f-b351d3228a1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81565b10-gwgm9\" (UID: \"b1655cc1-72e4-4f89-a69f-b351d3228a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9" Oct 30 22:25:55.152828 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.152743 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1655cc1-72e4-4f89-a69f-b351d3228a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81565b10-gwgm9\" (UID: \"b1655cc1-72e4-4f89-a69f-b351d3228a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9" Oct 30 22:25:55.153367 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.153344 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1655cc1-72e4-4f89-a69f-b351d3228a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81565b10-gwgm9\" (UID: \"b1655cc1-72e4-4f89-a69f-b351d3228a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9" Oct 30 22:25:55.155229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.155186 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1655cc1-72e4-4f89-a69f-b351d3228a1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81565b10-gwgm9\" (UID: \"b1655cc1-72e4-4f89-a69f-b351d3228a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9" Oct 30 22:25:55.191291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.191267 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb" Oct 30 22:25:55.214107 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.214077 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffddb\" (UniqueName: \"kubernetes.io/projected/b1655cc1-72e4-4f89-a69f-b351d3228a1c-kube-api-access-ffddb\") pod \"maas-api-revoke-sub-81565b10-gwgm9\" (UID: \"b1655cc1-72e4-4f89-a69f-b351d3228a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9" Oct 30 22:25:55.258929 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.258897 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9" Oct 30 22:25:55.358373 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:25:55.355016 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:25:55.594120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.594052 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb"] Oct 30 22:25:55.594240 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:25:55.594145 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d854e6_f96e_4c8f_906a_9f2a46eabc35.slice/crio-061b3a807cfe1712dd1ecea461cd14d4134659caae3ff44ab2b059bb2799270e WatchSource:0}: Error finding container 061b3a807cfe1712dd1ecea461cd14d4134659caae3ff44ab2b059bb2799270e: Status 404 returned error can't find the container with id 061b3a807cfe1712dd1ecea461cd14d4134659caae3ff44ab2b059bb2799270e Oct 30 22:25:55.709457 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.709427 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9"] Oct 30 22:25:55.774675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.772448 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" containerID="cri-o://b70397000885f2e87902708d6716e80ceee3b915224a6ef7189c5caa4fce504f" gracePeriod=45 Oct 30 22:25:55.786111 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.784976 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" containerID="cri-o://3e0e57c529d7f9d2c94b9bc88fa59fd02f35b9dc68c56e27b1688270be91c23b" gracePeriod=45 Oct 30 22:25:55.787733 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.787676 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": dial tcp 10.133.0.84:8000: connect: connection refused" Oct 30 22:25:55.796781 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.794510 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn"] Oct 30 22:25:55.806784 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.800793 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-bf7cbf-6rdvn"] Oct 30 22:25:55.806784 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:55.802953 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": read tcp 10.133.0.2:58316->10.133.0.80:8000: read: connection reset by peer" Oct 30 22:25:56.180399 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.180004 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:25:56.240225 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.240197 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:56.240974 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.240475 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:56.241659 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.241638 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:25:56.245732 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.245711 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:25:56.275172 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.275147 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-home\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-home\") pod \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " Oct 30 22:25:56.275388 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.275200 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-kube-api-access-74qh9\" (UniqueName: \"kubernetes.io/projected/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-kube-api-access-74qh9\") pod \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " Oct 30 22:25:56.275388 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.275291 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-tmp-dir\") pod \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " Oct 30 22:25:56.275524 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.275391 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-home" pod "c592bee9-4da3-43f3-9347-a8ee5dc1fce6" (UID: "c592bee9-4da3-43f3-9347-a8ee5dc1fce6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:56.275524 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.275402 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-dshm\") pod \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " Oct 30 22:25:56.275524 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.275458 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-tls-certs\" (UniqueName: \"kubernetes.io/secret/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-tls-certs\") pod \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " Oct 30 22:25:56.275731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.275524 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-tmp-dir" pod "c592bee9-4da3-43f3-9347-a8ee5dc1fce6" (UID: "c592bee9-4da3-43f3-9347-a8ee5dc1fce6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:56.275731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.275631 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-model-cache\") pod \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\" (UID: \"c592bee9-4da3-43f3-9347-a8ee5dc1fce6\") " Oct 30 22:25:56.275924 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.275901 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-model-cache" pod "c592bee9-4da3-43f3-9347-a8ee5dc1fce6" (UID: "c592bee9-4da3-43f3-9347-a8ee5dc1fce6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:56.276401 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.276377 4335 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-home\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:56.276524 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.276407 4335 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-tmp-dir\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:56.276524 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.276423 4335 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-model-cache\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:56.278013 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.277975 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-kube-api-access-74qh9" pod "c592bee9-4da3-43f3-9347-a8ee5dc1fce6" (UID: "c592bee9-4da3-43f3-9347-a8ee5dc1fce6"). InnerVolumeSpecName "kube-api-access-74qh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:56.278197 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.278170 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-tls-certs" pod "c592bee9-4da3-43f3-9347-a8ee5dc1fce6" (UID: "c592bee9-4da3-43f3-9347-a8ee5dc1fce6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:56.278297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.278242 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-dshm" pod "c592bee9-4da3-43f3-9347-a8ee5dc1fce6" (UID: "c592bee9-4da3-43f3-9347-a8ee5dc1fce6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:56.377661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.377626 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-model-cache\") pod \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " Oct 30 22:25:56.377852 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.377723 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-tls-certs\") pod \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " Oct 30 22:25:56.377852 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.377820 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-kube-api-access-8trz7\" (UniqueName: \"kubernetes.io/projected/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-kube-api-access-8trz7\") pod \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " Oct 30 22:25:56.377991 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.377872 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-model-cache" pod "a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" (UID: "a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:56.377991 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.377880 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-home\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-home\") pod \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " Oct 30 22:25:56.377991 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.377927 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-tmp-dir\") pod \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " Oct 30 22:25:56.378156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.378040 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-dshm\") pod \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\" (UID: \"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5\") " Oct 30 22:25:56.378231 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.378163 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-home" pod "a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" (UID: "a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:56.378287 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.378241 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-tmp-dir" pod "a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" (UID: "a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:56.378804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.378743 4335 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-tmp-dir\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:56.378804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.378772 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74qh9\" (UniqueName: \"kubernetes.io/projected/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-kube-api-access-74qh9\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:56.379023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.378875 4335 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-model-cache\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:56.379023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.378902 4335 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-dshm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:56.379023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.378918 4335 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c592bee9-4da3-43f3-9347-a8ee5dc1fce6-tls-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:56.379023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.378936 4335 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-home\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:56.380441 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.380412 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-dshm" pod "a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" (UID: "a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:56.380441 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.380430 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-tls-certs" pod "a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" (UID: "a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:56.380638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.380442 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-kube-api-access-8trz7" pod "a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" (UID: "a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5"). InnerVolumeSpecName "kube-api-access-8trz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:56.480965 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.480933 4335 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-dshm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:56.480965 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.480964 4335 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-tls-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:56.481158 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.480977 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8trz7\" (UniqueName: \"kubernetes.io/projected/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5-kube-api-access-8trz7\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:56.524099 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.524071 4335 generic.go:356] "Generic (PLEG): container finished" podID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerID="3e0e57c529d7f9d2c94b9bc88fa59fd02f35b9dc68c56e27b1688270be91c23b" exitCode=0 Oct 30 22:25:56.524278 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.524131 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" Oct 30 22:25:56.524278 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.524167 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" event={"ID":"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5","Type":"ContainerDied","Data":"3e0e57c529d7f9d2c94b9bc88fa59fd02f35b9dc68c56e27b1688270be91c23b"} Oct 30 22:25:56.524278 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.524226 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr" event={"ID":"a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5","Type":"ContainerDied","Data":"67e26428ec3ed711b3a82717bca06312d1f7af2ea20117b7f97df1b520bd645a"} Oct 30 22:25:56.524278 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.524248 4335 scope.go:122] "RemoveContainer" containerID="3e0e57c529d7f9d2c94b9bc88fa59fd02f35b9dc68c56e27b1688270be91c23b" Oct 30 22:25:56.533043 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.533014 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb" event={"ID":"97d854e6-f96e-4c8f-906a-9f2a46eabc35","Type":"ContainerStarted","Data":"882ece59432e9b9a14a7e6e0720296744ca8b67ce3ac7671d1ff8dfb292303a5"} Oct 30 22:25:56.533212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.533062 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb" event={"ID":"97d854e6-f96e-4c8f-906a-9f2a46eabc35","Type":"ContainerStarted","Data":"061b3a807cfe1712dd1ecea461cd14d4134659caae3ff44ab2b059bb2799270e"} Oct 30 22:25:56.535914 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.535824 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9" event={"ID":"b1655cc1-72e4-4f89-a69f-b351d3228a1c","Type":"ContainerStarted","Data":"e5a998d8202c0969d0a7ec9a9d31b8199d7b221b65089a29a932adb77e9e1785"} Oct 30 22:25:56.535914 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.535873 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9" event={"ID":"b1655cc1-72e4-4f89-a69f-b351d3228a1c","Type":"ContainerStarted","Data":"66520c0cccc3fa4c94f40bc50bc36a550c4495bed408be225e9650fe5e6ae053"} Oct 30 22:25:56.540663 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.540086 4335 generic.go:356] "Generic (PLEG): container finished" podID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerID="b70397000885f2e87902708d6716e80ceee3b915224a6ef7189c5caa4fce504f" exitCode=0 Oct 30 22:25:56.541050 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.540839 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" Oct 30 22:25:56.548027 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.547856 4335 scope.go:122] "RemoveContainer" containerID="3e0e57c529d7f9d2c94b9bc88fa59fd02f35b9dc68c56e27b1688270be91c23b" Oct 30 22:25:56.548027 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.547858 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" event={"ID":"c592bee9-4da3-43f3-9347-a8ee5dc1fce6","Type":"ContainerDied","Data":"b70397000885f2e87902708d6716e80ceee3b915224a6ef7189c5caa4fce504f"} Oct 30 22:25:56.548027 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.547950 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk" event={"ID":"c592bee9-4da3-43f3-9347-a8ee5dc1fce6","Type":"ContainerDied","Data":"643f3e7d1110fde588c851dbb86a803b380a108df6734187a85c65874ad2d94e"} Oct 30 22:25:56.548296 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:25:56.548244 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e0e57c529d7f9d2c94b9bc88fa59fd02f35b9dc68c56e27b1688270be91c23b\": container with ID starting with 3e0e57c529d7f9d2c94b9bc88fa59fd02f35b9dc68c56e27b1688270be91c23b not found: ID does not exist" containerID="3e0e57c529d7f9d2c94b9bc88fa59fd02f35b9dc68c56e27b1688270be91c23b" Oct 30 22:25:56.548296 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.548274 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e0e57c529d7f9d2c94b9bc88fa59fd02f35b9dc68c56e27b1688270be91c23b"} err="failed to get container status \"3e0e57c529d7f9d2c94b9bc88fa59fd02f35b9dc68c56e27b1688270be91c23b\": rpc error: code = NotFound desc = could not find container \"3e0e57c529d7f9d2c94b9bc88fa59fd02f35b9dc68c56e27b1688270be91c23b\": container with ID starting with 3e0e57c529d7f9d2c94b9bc88fa59fd02f35b9dc68c56e27b1688270be91c23b not found: ID does not exist" Oct 30 22:25:56.548428 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.548318 4335 scope.go:122] "RemoveContainer" containerID="b70397000885f2e87902708d6716e80ceee3b915224a6ef7189c5caa4fce504f" Oct 30 22:25:56.559759 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.559738 4335 scope.go:122] "RemoveContainer" containerID="b70397000885f2e87902708d6716e80ceee3b915224a6ef7189c5caa4fce504f" Oct 30 22:25:56.560064 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:25:56.560040 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70397000885f2e87902708d6716e80ceee3b915224a6ef7189c5caa4fce504f\": container with ID starting with b70397000885f2e87902708d6716e80ceee3b915224a6ef7189c5caa4fce504f not found: ID does not exist" containerID="b70397000885f2e87902708d6716e80ceee3b915224a6ef7189c5caa4fce504f" Oct 30 22:25:56.560064 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.560069 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70397000885f2e87902708d6716e80ceee3b915224a6ef7189c5caa4fce504f"} err="failed to get container status \"b70397000885f2e87902708d6716e80ceee3b915224a6ef7189c5caa4fce504f\": rpc error: code = NotFound desc = could not find container \"b70397000885f2e87902708d6716e80ceee3b915224a6ef7189c5caa4fce504f\": container with ID starting with b70397000885f2e87902708d6716e80ceee3b915224a6ef7189c5caa4fce504f not found: ID does not exist" Oct 30 22:25:56.671453 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.671345 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9" podStartSLOduration=2.671330351 podStartE2EDuration="2.671330351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:56.666947646 +0000 UTC m=+2489.767379393" watchObservedRunningTime="2026-10-30 22:25:56.671330351 +0000 UTC m=+2489.771762083" Oct 30 22:25:56.902490 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:56.902421 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb" podStartSLOduration=2.902402613 podStartE2EDuration="2.902402613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:56.898964853 +0000 UTC m=+2489.999396598" watchObservedRunningTime="2026-10-30 22:25:56.902402613 +0000 UTC m=+2490.002834346" Oct 30 22:25:57.024762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:57.024686 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr"] Oct 30 22:25:57.031654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:57.031605 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cef074/simulator-w3-cef074-kserve-57fdd9dd49-kkmgr"] Oct 30 22:25:57.063363 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:57.063324 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" path="/var/lib/kubelet/pods/a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5/volumes" Oct 30 22:25:57.064127 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:57.064103 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acc37148-c569-4d8d-92ff-6a5bd1776aed" path="/var/lib/kubelet/pods/acc37148-c569-4d8d-92ff-6a5bd1776aed/volumes" Oct 30 22:25:57.131040 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:57.131011 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk"] Oct 30 22:25:57.137402 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:57.137379 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cef074/premium-w3-cef074-kserve-6899757bb8-zx7hk"] Oct 30 22:25:57.437755 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:57.437704 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" podUID="0b00cf8d-cd32-402b-8815-0c239365c8da" containerName="payload-pre-processing" containerID="cri-o://0c98e66e4b934461163e898748e1ae16e7135859a309098fa95feb09ba51d51c" gracePeriod=25 Oct 30 22:25:57.559526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:57.559492 4335 generic.go:356] "Generic (PLEG): container finished" podID="97d854e6-f96e-4c8f-906a-9f2a46eabc35" containerID="882ece59432e9b9a14a7e6e0720296744ca8b67ce3ac7671d1ff8dfb292303a5" exitCode=0 Oct 30 22:25:57.562020 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:57.559584 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb" event={"ID":"97d854e6-f96e-4c8f-906a-9f2a46eabc35","Type":"ContainerDied","Data":"882ece59432e9b9a14a7e6e0720296744ca8b67ce3ac7671d1ff8dfb292303a5"} Oct 30 22:25:57.562020 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:57.561473 4335 generic.go:356] "Generic (PLEG): container finished" podID="b1655cc1-72e4-4f89-a69f-b351d3228a1c" containerID="e5a998d8202c0969d0a7ec9a9d31b8199d7b221b65089a29a932adb77e9e1785" exitCode=0 Oct 30 22:25:57.562020 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:57.561712 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9" event={"ID":"b1655cc1-72e4-4f89-a69f-b351d3228a1c","Type":"ContainerDied","Data":"e5a998d8202c0969d0a7ec9a9d31b8199d7b221b65089a29a932adb77e9e1785"} Oct 30 22:25:58.750670 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:58.750605 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9" Oct 30 22:25:58.781392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:58.781370 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb" Oct 30 22:25:58.812496 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:58.812455 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1655cc1-72e4-4f89-a69f-b351d3228a1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1655cc1-72e4-4f89-a69f-b351d3228a1c-maas-api-xks-ca\") pod \"b1655cc1-72e4-4f89-a69f-b351d3228a1c\" (UID: \"b1655cc1-72e4-4f89-a69f-b351d3228a1c\") " Oct 30 22:25:58.812663 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:58.812637 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1655cc1-72e4-4f89-a69f-b351d3228a1c-kube-api-access-ffddb\" (UniqueName: \"kubernetes.io/projected/b1655cc1-72e4-4f89-a69f-b351d3228a1c-kube-api-access-ffddb\") pod \"b1655cc1-72e4-4f89-a69f-b351d3228a1c\" (UID: \"b1655cc1-72e4-4f89-a69f-b351d3228a1c\") " Oct 30 22:25:58.812808 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:58.812789 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1655cc1-72e4-4f89-a69f-b351d3228a1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1655cc1-72e4-4f89-a69f-b351d3228a1c-maas-api-openshift-ca\") pod \"b1655cc1-72e4-4f89-a69f-b351d3228a1c\" (UID: \"b1655cc1-72e4-4f89-a69f-b351d3228a1c\") " Oct 30 22:25:58.813416 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:58.813394 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1655cc1-72e4-4f89-a69f-b351d3228a1c-maas-api-openshift-ca" pod "b1655cc1-72e4-4f89-a69f-b351d3228a1c" (UID: "b1655cc1-72e4-4f89-a69f-b351d3228a1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:58.815218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:58.815197 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1655cc1-72e4-4f89-a69f-b351d3228a1c-maas-api-xks-ca" pod "b1655cc1-72e4-4f89-a69f-b351d3228a1c" (UID: "b1655cc1-72e4-4f89-a69f-b351d3228a1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:58.815413 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:58.815399 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1655cc1-72e4-4f89-a69f-b351d3228a1c-kube-api-access-ffddb" pod "b1655cc1-72e4-4f89-a69f-b351d3228a1c" (UID: "b1655cc1-72e4-4f89-a69f-b351d3228a1c"). InnerVolumeSpecName "kube-api-access-ffddb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:58.916928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:58.916890 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97d854e6-f96e-4c8f-906a-9f2a46eabc35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d854e6-f96e-4c8f-906a-9f2a46eabc35-maas-api-xks-ca\") pod \"97d854e6-f96e-4c8f-906a-9f2a46eabc35\" (UID: \"97d854e6-f96e-4c8f-906a-9f2a46eabc35\") " Oct 30 22:25:58.917141 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:58.917104 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97d854e6-f96e-4c8f-906a-9f2a46eabc35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d854e6-f96e-4c8f-906a-9f2a46eabc35-maas-api-openshift-ca\") pod \"97d854e6-f96e-4c8f-906a-9f2a46eabc35\" (UID: \"97d854e6-f96e-4c8f-906a-9f2a46eabc35\") " Oct 30 22:25:58.917572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:58.917513 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97d854e6-f96e-4c8f-906a-9f2a46eabc35-maas-api-openshift-ca" pod "97d854e6-f96e-4c8f-906a-9f2a46eabc35" (UID: "97d854e6-f96e-4c8f-906a-9f2a46eabc35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:58.921544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:58.918074 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97d854e6-f96e-4c8f-906a-9f2a46eabc35-kube-api-access-sjkds\" (UniqueName: \"kubernetes.io/projected/97d854e6-f96e-4c8f-906a-9f2a46eabc35-kube-api-access-sjkds\") pod \"97d854e6-f96e-4c8f-906a-9f2a46eabc35\" (UID: \"97d854e6-f96e-4c8f-906a-9f2a46eabc35\") " Oct 30 22:25:58.921544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:58.920836 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d854e6-f96e-4c8f-906a-9f2a46eabc35-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:58.921544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:58.920856 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffddb\" (UniqueName: \"kubernetes.io/projected/b1655cc1-72e4-4f89-a69f-b351d3228a1c-kube-api-access-ffddb\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:58.921544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:58.920865 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1655cc1-72e4-4f89-a69f-b351d3228a1c-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:58.921544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:58.920874 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1655cc1-72e4-4f89-a69f-b351d3228a1c-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:58.922121 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:58.921950 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97d854e6-f96e-4c8f-906a-9f2a46eabc35-maas-api-xks-ca" pod "97d854e6-f96e-4c8f-906a-9f2a46eabc35" (UID: "97d854e6-f96e-4c8f-906a-9f2a46eabc35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:58.924306 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:58.924034 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d854e6-f96e-4c8f-906a-9f2a46eabc35-kube-api-access-sjkds" pod "97d854e6-f96e-4c8f-906a-9f2a46eabc35" (UID: "97d854e6-f96e-4c8f-906a-9f2a46eabc35"). InnerVolumeSpecName "kube-api-access-sjkds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:59.022855 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:59.022820 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d854e6-f96e-4c8f-906a-9f2a46eabc35-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:59.022855 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:59.022859 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjkds\" (UniqueName: \"kubernetes.io/projected/97d854e6-f96e-4c8f-906a-9f2a46eabc35-kube-api-access-sjkds\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:59.058815 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:59.058777 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" path="/var/lib/kubelet/pods/c592bee9-4da3-43f3-9347-a8ee5dc1fce6/volumes" Oct 30 22:25:59.579324 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:59.579282 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb" Oct 30 22:25:59.579324 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:59.579311 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb" event={"ID":"97d854e6-f96e-4c8f-906a-9f2a46eabc35","Type":"ContainerDied","Data":"061b3a807cfe1712dd1ecea461cd14d4134659caae3ff44ab2b059bb2799270e"} Oct 30 22:25:59.579564 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:59.579365 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="061b3a807cfe1712dd1ecea461cd14d4134659caae3ff44ab2b059bb2799270e" Oct 30 22:25:59.581116 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:59.581085 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9" Oct 30 22:25:59.581250 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:59.581125 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9" event={"ID":"b1655cc1-72e4-4f89-a69f-b351d3228a1c","Type":"ContainerDied","Data":"66520c0cccc3fa4c94f40bc50bc36a550c4495bed408be225e9650fe5e6ae053"} Oct 30 22:25:59.581250 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:25:59.581172 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66520c0cccc3fa4c94f40bc50bc36a550c4495bed408be225e9650fe5e6ae053" Oct 30 22:26:01.389158 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.389120 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" containerID="cri-o://c0b4fd4aba13d721c0a9c6bdcf4c332776507182489fa495b764117a78901a87" gracePeriod=45 Oct 30 22:26:01.437362 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.437330 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" containerID="cri-o://23faa20178757dd3234fef0531352b0869bc850769d0a3cae07e081ef4ce0157" gracePeriod=45 Oct 30 22:26:01.602293 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.602262 4335 generic.go:356] "Generic (PLEG): container finished" podID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerID="c0b4fd4aba13d721c0a9c6bdcf4c332776507182489fa495b764117a78901a87" exitCode=0 Oct 30 22:26:01.602489 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.602341 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" event={"ID":"a4bee3da-5d5b-4663-9f31-bf0b2388e06e","Type":"ContainerDied","Data":"c0b4fd4aba13d721c0a9c6bdcf4c332776507182489fa495b764117a78901a87"} Oct 30 22:26:01.604501 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.604478 4335 generic.go:356] "Generic (PLEG): container finished" podID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerID="23faa20178757dd3234fef0531352b0869bc850769d0a3cae07e081ef4ce0157" exitCode=0 Oct 30 22:26:01.604690 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.604558 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" event={"ID":"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23","Type":"ContainerDied","Data":"23faa20178757dd3234fef0531352b0869bc850769d0a3cae07e081ef4ce0157"} Oct 30 22:26:01.782455 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.782430 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:26:01.841008 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.840984 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:26:01.867110 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.867085 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-model-cache\") pod \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " Oct 30 22:26:01.867229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.867141 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-dshm\") pod \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " Oct 30 22:26:01.867229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.867200 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-tls-certs\" (UniqueName: \"kubernetes.io/secret/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-tls-certs\") pod \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " Oct 30 22:26:01.867327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.867315 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-home\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-home\") pod \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " Oct 30 22:26:01.867371 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.867319 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-model-cache" pod "a4bee3da-5d5b-4663-9f31-bf0b2388e06e" (UID: "a4bee3da-5d5b-4663-9f31-bf0b2388e06e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:01.867422 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.867406 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-tmp-dir\") pod \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " Oct 30 22:26:01.867465 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.867448 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-kube-api-access-g9tnv\" (UniqueName: \"kubernetes.io/projected/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-kube-api-access-g9tnv\") pod \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\" (UID: \"a4bee3da-5d5b-4663-9f31-bf0b2388e06e\") " Oct 30 22:26:01.867655 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.867578 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-home" pod "a4bee3da-5d5b-4663-9f31-bf0b2388e06e" (UID: "a4bee3da-5d5b-4663-9f31-bf0b2388e06e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:01.867759 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.867692 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-tmp-dir" pod "a4bee3da-5d5b-4663-9f31-bf0b2388e06e" (UID: "a4bee3da-5d5b-4663-9f31-bf0b2388e06e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:01.868298 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.868250 4335 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-model-cache\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:01.868397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.868300 4335 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-home\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:01.868397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.868340 4335 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-tmp-dir\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:01.869819 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.869795 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-kube-api-access-g9tnv" pod "a4bee3da-5d5b-4663-9f31-bf0b2388e06e" (UID: "a4bee3da-5d5b-4663-9f31-bf0b2388e06e"). InnerVolumeSpecName "kube-api-access-g9tnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:01.870000 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.869975 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-tls-certs" pod "a4bee3da-5d5b-4663-9f31-bf0b2388e06e" (UID: "a4bee3da-5d5b-4663-9f31-bf0b2388e06e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:01.870000 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.869982 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-dshm" pod "a4bee3da-5d5b-4663-9f31-bf0b2388e06e" (UID: "a4bee3da-5d5b-4663-9f31-bf0b2388e06e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:01.969691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.969662 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-kube-api-access-4p7sl\" (UniqueName: \"kubernetes.io/projected/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-kube-api-access-4p7sl\") pod \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " Oct 30 22:26:01.969866 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.969711 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-home\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-home\") pod \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " Oct 30 22:26:01.969931 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.969899 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-model-cache\") pod \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " Oct 30 22:26:01.969987 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.969962 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-tls-certs\") pod \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " Oct 30 22:26:01.970048 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.970001 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-dshm\") pod \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " Oct 30 22:26:01.970102 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.970046 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-home" pod "4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" (UID: "4eefc8ca-80bf-4757-91ff-ef5d8ebffa23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:01.970102 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.970053 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-tmp-dir\") pod \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\" (UID: \"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23\") " Oct 30 22:26:01.970263 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.970241 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-tmp-dir" pod "4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" (UID: "4eefc8ca-80bf-4757-91ff-ef5d8ebffa23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:01.970521 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.970487 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-model-cache" pod "4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" (UID: "4eefc8ca-80bf-4757-91ff-ef5d8ebffa23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:01.970780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.970759 4335 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-tmp-dir\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:01.970906 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.970784 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9tnv\" (UniqueName: \"kubernetes.io/projected/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-kube-api-access-g9tnv\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:01.970906 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.970801 4335 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-home\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:01.970906 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.970815 4335 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-dshm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:01.970906 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.970827 4335 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4bee3da-5d5b-4663-9f31-bf0b2388e06e-tls-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:01.970906 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.970840 4335 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-model-cache\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:01.972693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.972601 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-dshm" pod "4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" (UID: "4eefc8ca-80bf-4757-91ff-ef5d8ebffa23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:01.973240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.973217 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-kube-api-access-4p7sl" pod "4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" (UID: "4eefc8ca-80bf-4757-91ff-ef5d8ebffa23"). InnerVolumeSpecName "kube-api-access-4p7sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:01.973347 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:01.973242 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-tls-certs" pod "4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" (UID: "4eefc8ca-80bf-4757-91ff-ef5d8ebffa23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:02.072498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:02.072465 4335 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-tls-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:02.072498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:02.072500 4335 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-dshm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:02.072737 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:02.072516 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p7sl\" (UniqueName: \"kubernetes.io/projected/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23-kube-api-access-4p7sl\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:02.614981 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:02.614947 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" Oct 30 22:26:02.614981 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:02.614968 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf" event={"ID":"a4bee3da-5d5b-4663-9f31-bf0b2388e06e","Type":"ContainerDied","Data":"17433e435513640b9da61af3e43f759c579bfa829903af95a373e4bf92f73327"} Oct 30 22:26:02.615496 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:02.615024 4335 scope.go:122] "RemoveContainer" containerID="c0b4fd4aba13d721c0a9c6bdcf4c332776507182489fa495b764117a78901a87" Oct 30 22:26:02.616937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:02.616909 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" event={"ID":"4eefc8ca-80bf-4757-91ff-ef5d8ebffa23","Type":"ContainerDied","Data":"1d55a0e6d60d1dc21a7372b6e93fd86709c14b37f012ef1c80397a00226b526b"} Oct 30 22:26:02.616937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:02.616937 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd" Oct 30 22:26:02.628821 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:02.627984 4335 scope.go:122] "RemoveContainer" containerID="23faa20178757dd3234fef0531352b0869bc850769d0a3cae07e081ef4ce0157" Oct 30 22:26:02.791562 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:02.791443 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd"] Oct 30 22:26:02.831697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:02.831663 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-a7b06b/distinct-w5-a7b06b-kserve-66b8b77ff8-trqqd"] Oct 30 22:26:02.896176 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:02.896145 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf"] Oct 30 22:26:02.903333 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:02.903308 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-a7b06b/distinct-2-w5-a7b06b-kserve-56ffd6db9-2jggf"] Oct 30 22:26:02.972653 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:02.972607 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" podUID="0b00cf8d-cd32-402b-8815-0c239365c8da" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:26:03.000299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:03.000278 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n"] Oct 30 22:26:03.005093 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:03.005073 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25237a-fgv6n"] Oct 30 22:26:03.057965 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:03.057939 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" path="/var/lib/kubelet/pods/4eefc8ca-80bf-4757-91ff-ef5d8ebffa23/volumes" Oct 30 22:26:03.058429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:03.058414 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8" path="/var/lib/kubelet/pods/5ddba4b4-abbc-472e-8b4f-2ea2aa876ed8/volumes" Oct 30 22:26:03.058862 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:03.058848 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" path="/var/lib/kubelet/pods/a4bee3da-5d5b-4663-9f31-bf0b2388e06e/volumes" Oct 30 22:26:04.655289 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:04.655259 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p"] Oct 30 22:26:04.655818 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:04.655496 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" podUID="60a004f7-2dc7-4b0e-a844-48404c0e35f2" containerName="istio-proxy" containerID="cri-o://e8df66045448c86fea758dc8bdf2cfedf466d46e4c4df435f9b958c926602c1c" gracePeriod=30 Oct 30 22:26:04.702409 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:04.702382 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996"] Oct 30 22:26:04.702652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:04.702591 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" podUID="0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" containerName="istio-proxy" containerID="cri-o://85e87dce410c5854a136b52a2b53bef92c60b8345bd575d338a28aa60dd02927" gracePeriod=30 Oct 30 22:26:06.226852 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:06.226820 4335 patch_prober.go:29] interesting pod/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:26:06.227337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:06.226873 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" podUID="0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:26:10.023994 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.023960 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:26:10.045237 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.045212 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 22:26:10.067748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.067719 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-kube-api-access-k58tk\" (UniqueName: \"kubernetes.io/projected/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-kube-api-access-k58tk\") pod \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " Oct 30 22:26:10.067873 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.067768 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-podinfo\") pod \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " Oct 30 22:26:10.067873 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.067799 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-podinfo\") pod \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " Oct 30 22:26:10.067873 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.067838 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-workload-certs\") pod \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " Oct 30 22:26:10.067873 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.067869 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istiod-ca-cert\") pod \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " Oct 30 22:26:10.068097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.067921 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a004f7-2dc7-4b0e-a844-48404c0e35f2-kube-api-access-t2ss6\" (UniqueName: \"kubernetes.io/projected/60a004f7-2dc7-4b0e-a844-48404c0e35f2-kube-api-access-t2ss6\") pod \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " Oct 30 22:26:10.068097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.067962 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-credential-socket\") pod \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " Oct 30 22:26:10.068097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.068007 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-data\") pod \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " Oct 30 22:26:10.068097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.068042 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istiod-ca-cert\") pod \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " Oct 30 22:26:10.068307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.068116 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-workload-certs\") pod \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " Oct 30 22:26:10.068307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.068146 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-token\" (UniqueName: \"kubernetes.io/projected/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-token\") pod \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " Oct 30 22:26:10.068307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.068162 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-workload-certs" pod "0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" (UID: "0f7b76ed-f54c-4ff1-930a-c5f183bc72ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:10.068307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.068218 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-envoy\") pod \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " Oct 30 22:26:10.068307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.068283 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-token\" (UniqueName: \"kubernetes.io/projected/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-token\") pod \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " Oct 30 22:26:10.068574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.068315 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-credential-socket\") pod \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " Oct 30 22:26:10.068574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.068376 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-data\") pod \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " Oct 30 22:26:10.068574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.068386 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-credential-socket" pod "60a004f7-2dc7-4b0e-a844-48404c0e35f2" (UID: "60a004f7-2dc7-4b0e-a844-48404c0e35f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:10.068574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.068411 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-workload-socket\") pod \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\" (UID: \"60a004f7-2dc7-4b0e-a844-48404c0e35f2\") " Oct 30 22:26:10.068574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.068451 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-workload-socket\") pod \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " Oct 30 22:26:10.068574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.068486 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-envoy\") pod \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\" (UID: \"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec\") " Oct 30 22:26:10.068941 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.068703 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istiod-ca-cert" pod "60a004f7-2dc7-4b0e-a844-48404c0e35f2" (UID: "60a004f7-2dc7-4b0e-a844-48404c0e35f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:10.069047 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.069026 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.069114 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.069052 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.069114 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.069069 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.069747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.069720 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istiod-ca-cert" pod "0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" (UID: "0f7b76ed-f54c-4ff1-930a-c5f183bc72ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:10.069986 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.069966 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-data" pod "60a004f7-2dc7-4b0e-a844-48404c0e35f2" (UID: "60a004f7-2dc7-4b0e-a844-48404c0e35f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:10.070184 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.070165 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-data" pod "0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" (UID: "0f7b76ed-f54c-4ff1-930a-c5f183bc72ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:10.070502 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.070475 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-kube-api-access-k58tk" pod "0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" (UID: "0f7b76ed-f54c-4ff1-930a-c5f183bc72ec"). InnerVolumeSpecName "kube-api-access-k58tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:10.070812 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.070711 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-podinfo" pod "0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" (UID: "0f7b76ed-f54c-4ff1-930a-c5f183bc72ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:10.070812 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.070719 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-workload-socket" pod "60a004f7-2dc7-4b0e-a844-48404c0e35f2" (UID: "60a004f7-2dc7-4b0e-a844-48404c0e35f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:10.070812 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.070764 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-podinfo" pod "60a004f7-2dc7-4b0e-a844-48404c0e35f2" (UID: "60a004f7-2dc7-4b0e-a844-48404c0e35f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:10.071090 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.071033 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-credential-socket" pod "0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" (UID: "0f7b76ed-f54c-4ff1-930a-c5f183bc72ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:10.071509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.071210 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-workload-socket" pod "0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" (UID: "0f7b76ed-f54c-4ff1-930a-c5f183bc72ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:10.071509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.071440 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-envoy" pod "0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" (UID: "0f7b76ed-f54c-4ff1-930a-c5f183bc72ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:10.071509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.071458 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-workload-certs" pod "60a004f7-2dc7-4b0e-a844-48404c0e35f2" (UID: "60a004f7-2dc7-4b0e-a844-48404c0e35f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:10.072254 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.072228 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a004f7-2dc7-4b0e-a844-48404c0e35f2-kube-api-access-t2ss6" pod "60a004f7-2dc7-4b0e-a844-48404c0e35f2" (UID: "60a004f7-2dc7-4b0e-a844-48404c0e35f2"). InnerVolumeSpecName "kube-api-access-t2ss6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:10.072888 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.072866 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-token" pod "0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" (UID: "0f7b76ed-f54c-4ff1-930a-c5f183bc72ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:10.073013 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.072993 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-token" pod "60a004f7-2dc7-4b0e-a844-48404c0e35f2" (UID: "60a004f7-2dc7-4b0e-a844-48404c0e35f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:10.073307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.073289 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-envoy" pod "60a004f7-2dc7-4b0e-a844-48404c0e35f2" (UID: "60a004f7-2dc7-4b0e-a844-48404c0e35f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:10.170686 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.170658 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.170686 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.170684 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.170910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.170694 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.170910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.170705 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.170910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.170714 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k58tk\" (UniqueName: \"kubernetes.io/projected/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-kube-api-access-k58tk\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.170910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.170724 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.170910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.170732 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.170910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.170740 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2ss6\" (UniqueName: \"kubernetes.io/projected/60a004f7-2dc7-4b0e-a844-48404c0e35f2-kube-api-access-t2ss6\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.170910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.170748 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.170910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.170757 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.170910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.170766 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.170910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.170774 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.170910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.170782 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.170910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.170789 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a004f7-2dc7-4b0e-a844-48404c0e35f2-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.170910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.170798 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.688105 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.688068 4335 generic.go:356] "Generic (PLEG): container finished" podID="60a004f7-2dc7-4b0e-a844-48404c0e35f2" containerID="e8df66045448c86fea758dc8bdf2cfedf466d46e4c4df435f9b958c926602c1c" exitCode=0 Oct 30 22:26:10.688297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.688116 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" Oct 30 22:26:10.688297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.688152 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" event={"ID":"60a004f7-2dc7-4b0e-a844-48404c0e35f2","Type":"ContainerDied","Data":"e8df66045448c86fea758dc8bdf2cfedf466d46e4c4df435f9b958c926602c1c"} Oct 30 22:26:10.688297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.688213 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p" event={"ID":"60a004f7-2dc7-4b0e-a844-48404c0e35f2","Type":"ContainerDied","Data":"64d89ebbefbf26bb607fea4029024203d669c5343ddf0d331abfa821c3f40409"} Oct 30 22:26:10.688297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.688245 4335 scope.go:122] "RemoveContainer" containerID="e8df66045448c86fea758dc8bdf2cfedf466d46e4c4df435f9b958c926602c1c" Oct 30 22:26:10.696692 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.696592 4335 generic.go:356] "Generic (PLEG): container finished" podID="0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" containerID="85e87dce410c5854a136b52a2b53bef92c60b8345bd575d338a28aa60dd02927" exitCode=0 Oct 30 22:26:10.696692 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.696650 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" event={"ID":"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec","Type":"ContainerDied","Data":"85e87dce410c5854a136b52a2b53bef92c60b8345bd575d338a28aa60dd02927"} Oct 30 22:26:10.696861 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.696701 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" Oct 30 22:26:10.696861 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.696712 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996" event={"ID":"0f7b76ed-f54c-4ff1-930a-c5f183bc72ec","Type":"ContainerDied","Data":"ce45e0851ece194cd868bf3bc60915f94dcbed0a5451c98026a614fdf600aed5"} Oct 30 22:26:10.702937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.702920 4335 scope.go:122] "RemoveContainer" containerID="e8df66045448c86fea758dc8bdf2cfedf466d46e4c4df435f9b958c926602c1c" Oct 30 22:26:10.703153 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:26:10.703134 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8df66045448c86fea758dc8bdf2cfedf466d46e4c4df435f9b958c926602c1c\": container with ID starting with e8df66045448c86fea758dc8bdf2cfedf466d46e4c4df435f9b958c926602c1c not found: ID does not exist" containerID="e8df66045448c86fea758dc8bdf2cfedf466d46e4c4df435f9b958c926602c1c" Oct 30 22:26:10.703219 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.703159 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8df66045448c86fea758dc8bdf2cfedf466d46e4c4df435f9b958c926602c1c"} err="failed to get container status \"e8df66045448c86fea758dc8bdf2cfedf466d46e4c4df435f9b958c926602c1c\": rpc error: code = NotFound desc = could not find container \"e8df66045448c86fea758dc8bdf2cfedf466d46e4c4df435f9b958c926602c1c\": container with ID starting with e8df66045448c86fea758dc8bdf2cfedf466d46e4c4df435f9b958c926602c1c not found: ID does not exist" Oct 30 22:26:10.703219 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.703174 4335 scope.go:122] "RemoveContainer" containerID="85e87dce410c5854a136b52a2b53bef92c60b8345bd575d338a28aa60dd02927" Oct 30 22:26:10.713582 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.713561 4335 scope.go:122] "RemoveContainer" containerID="1b75afff3530fae1c95aa5cd6755bd7ddbdf89a857a28a58ae17a03cb2984e85" Oct 30 22:26:10.723053 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.723038 4335 scope.go:122] "RemoveContainer" containerID="85e87dce410c5854a136b52a2b53bef92c60b8345bd575d338a28aa60dd02927" Oct 30 22:26:10.723333 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:26:10.723314 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e87dce410c5854a136b52a2b53bef92c60b8345bd575d338a28aa60dd02927\": container with ID starting with 85e87dce410c5854a136b52a2b53bef92c60b8345bd575d338a28aa60dd02927 not found: ID does not exist" containerID="85e87dce410c5854a136b52a2b53bef92c60b8345bd575d338a28aa60dd02927" Oct 30 22:26:10.723391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.723340 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e87dce410c5854a136b52a2b53bef92c60b8345bd575d338a28aa60dd02927"} err="failed to get container status \"85e87dce410c5854a136b52a2b53bef92c60b8345bd575d338a28aa60dd02927\": rpc error: code = NotFound desc = could not find container \"85e87dce410c5854a136b52a2b53bef92c60b8345bd575d338a28aa60dd02927\": container with ID starting with 85e87dce410c5854a136b52a2b53bef92c60b8345bd575d338a28aa60dd02927 not found: ID does not exist" Oct 30 22:26:10.723391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.723356 4335 scope.go:122] "RemoveContainer" containerID="1b75afff3530fae1c95aa5cd6755bd7ddbdf89a857a28a58ae17a03cb2984e85" Oct 30 22:26:10.723589 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:26:10.723571 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b75afff3530fae1c95aa5cd6755bd7ddbdf89a857a28a58ae17a03cb2984e85\": container with ID starting with 1b75afff3530fae1c95aa5cd6755bd7ddbdf89a857a28a58ae17a03cb2984e85 not found: ID does not exist" containerID="1b75afff3530fae1c95aa5cd6755bd7ddbdf89a857a28a58ae17a03cb2984e85" Oct 30 22:26:10.723669 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.723605 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b75afff3530fae1c95aa5cd6755bd7ddbdf89a857a28a58ae17a03cb2984e85"} err="failed to get container status \"1b75afff3530fae1c95aa5cd6755bd7ddbdf89a857a28a58ae17a03cb2984e85\": rpc error: code = NotFound desc = could not find container \"1b75afff3530fae1c95aa5cd6755bd7ddbdf89a857a28a58ae17a03cb2984e85\": container with ID starting with 1b75afff3530fae1c95aa5cd6755bd7ddbdf89a857a28a58ae17a03cb2984e85 not found: ID does not exist" Oct 30 22:26:10.854113 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.854091 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996"] Oct 30 22:26:10.857772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.857752 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-6d996"] Oct 30 22:26:10.966418 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.966346 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p"] Oct 30 22:26:10.973180 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:10.973150 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bf7cbf-openshift-default-854dc59b99-ffh8p"] Oct 30 22:26:11.057624 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:11.057584 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" path="/var/lib/kubelet/pods/0f7b76ed-f54c-4ff1-930a-c5f183bc72ec/volumes" Oct 30 22:26:11.058081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:11.058065 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a004f7-2dc7-4b0e-a844-48404c0e35f2" path="/var/lib/kubelet/pods/60a004f7-2dc7-4b0e-a844-48404c0e35f2/volumes" Oct 30 22:26:12.973601 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:12.973569 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" podUID="0b00cf8d-cd32-402b-8815-0c239365c8da" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:26:14.751085 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:14.751052 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9"] Oct 30 22:26:14.799891 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:14.799863 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1024a-z54t9"] Oct 30 22:26:15.058018 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:15.057992 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd15165-4075-470b-92ab-8246e2fc26b4" path="/var/lib/kubelet/pods/4dd15165-4075-470b-92ab-8246e2fc26b4/volumes" Oct 30 22:26:17.618312 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:17.618279 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" Oct 30 22:26:17.658645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:17.658587 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b00cf8d-cd32-402b-8815-0c239365c8da-kube-api-access-5wl8d\" (UniqueName: \"kubernetes.io/projected/0b00cf8d-cd32-402b-8815-0c239365c8da-kube-api-access-5wl8d\") pod \"0b00cf8d-cd32-402b-8815-0c239365c8da\" (UID: \"0b00cf8d-cd32-402b-8815-0c239365c8da\") " Oct 30 22:26:17.658804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:17.658727 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b00cf8d-cd32-402b-8815-0c239365c8da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b00cf8d-cd32-402b-8815-0c239365c8da-plugins-config-volume\") pod \"0b00cf8d-cd32-402b-8815-0c239365c8da\" (UID: \"0b00cf8d-cd32-402b-8815-0c239365c8da\") " Oct 30 22:26:17.659126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:17.659093 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b00cf8d-cd32-402b-8815-0c239365c8da-plugins-config-volume" pod "0b00cf8d-cd32-402b-8815-0c239365c8da" (UID: "0b00cf8d-cd32-402b-8815-0c239365c8da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:17.660992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:17.660949 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b00cf8d-cd32-402b-8815-0c239365c8da-kube-api-access-5wl8d" pod "0b00cf8d-cd32-402b-8815-0c239365c8da" (UID: "0b00cf8d-cd32-402b-8815-0c239365c8da"). InnerVolumeSpecName "kube-api-access-5wl8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:17.755998 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:17.755929 4335 generic.go:356] "Generic (PLEG): container finished" podID="0b00cf8d-cd32-402b-8815-0c239365c8da" containerID="0c98e66e4b934461163e898748e1ae16e7135859a309098fa95feb09ba51d51c" exitCode=0 Oct 30 22:26:17.756228 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:17.756210 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" event={"ID":"0b00cf8d-cd32-402b-8815-0c239365c8da","Type":"ContainerDied","Data":"0c98e66e4b934461163e898748e1ae16e7135859a309098fa95feb09ba51d51c"} Oct 30 22:26:17.756275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:17.756263 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" event={"ID":"0b00cf8d-cd32-402b-8815-0c239365c8da","Type":"ContainerDied","Data":"89c7fe63c2beae7d5ffe2b267627633224525546cc08d3b3e4b3ecb56ef9287c"} Oct 30 22:26:17.756309 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:17.756290 4335 scope.go:122] "RemoveContainer" containerID="0c98e66e4b934461163e898748e1ae16e7135859a309098fa95feb09ba51d51c" Oct 30 22:26:17.756453 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:17.756437 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh" Oct 30 22:26:17.760085 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:17.760059 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wl8d\" (UniqueName: \"kubernetes.io/projected/0b00cf8d-cd32-402b-8815-0c239365c8da-kube-api-access-5wl8d\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:17.760085 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:17.760084 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b00cf8d-cd32-402b-8815-0c239365c8da-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:17.771605 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:17.771585 4335 scope.go:122] "RemoveContainer" containerID="0c98e66e4b934461163e898748e1ae16e7135859a309098fa95feb09ba51d51c" Oct 30 22:26:17.771956 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:26:17.771931 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c98e66e4b934461163e898748e1ae16e7135859a309098fa95feb09ba51d51c\": container with ID starting with 0c98e66e4b934461163e898748e1ae16e7135859a309098fa95feb09ba51d51c not found: ID does not exist" containerID="0c98e66e4b934461163e898748e1ae16e7135859a309098fa95feb09ba51d51c" Oct 30 22:26:17.772064 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:17.771963 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c98e66e4b934461163e898748e1ae16e7135859a309098fa95feb09ba51d51c"} err="failed to get container status \"0c98e66e4b934461163e898748e1ae16e7135859a309098fa95feb09ba51d51c\": rpc error: code = NotFound desc = could not find container \"0c98e66e4b934461163e898748e1ae16e7135859a309098fa95feb09ba51d51c\": container with ID starting with 0c98e66e4b934461163e898748e1ae16e7135859a309098fa95feb09ba51d51c not found: ID does not exist" Oct 30 22:26:17.923088 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:17.923057 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh"] Oct 30 22:26:17.926718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:17.926687 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a7b06b-c5644847f-jw2sh"] Oct 30 22:26:19.057830 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:19.057789 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b00cf8d-cd32-402b-8815-0c239365c8da" path="/var/lib/kubelet/pods/0b00cf8d-cd32-402b-8815-0c239365c8da/volumes" Oct 30 22:26:20.102156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:20.102120 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6"] Oct 30 22:26:20.102743 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:20.102331 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" podUID="58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" containerName="istio-proxy" containerID="cri-o://9b99750cf13046cc3352d003c8e66342d562e720518a4e3f378c931820fec202" gracePeriod=30 Oct 30 22:26:21.798967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:21.798932 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn"] Oct 30 22:26:21.801494 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:21.801463 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" podUID="5e3c2933-2586-4001-aa99-c18c1a6d44bb" containerName="istio-proxy" containerID="cri-o://11fce6f669b7c04c675f8f922eb71fcc58e0adfab32b23bbaa5fa495eebfa8f1" gracePeriod=30 Oct 30 22:26:22.625833 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.625794 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq"] Oct 30 22:26:22.626731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626711 4335 cpu_manager.go:414] "containerMap: removing container" podUID="0b00cf8d-cd32-402b-8815-0c239365c8da" containerName="payload-pre-processing" Oct 30 22:26:22.626777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626735 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b00cf8d-cd32-402b-8815-0c239365c8da" containerName="payload-pre-processing" Oct 30 22:26:22.626777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626757 4335 cpu_manager.go:414] "containerMap: removing container" podUID="0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" containerName="istio-proxy" Oct 30 22:26:22.626777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626765 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" containerName="istio-proxy" Oct 30 22:26:22.626884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626778 4335 cpu_manager.go:414] "containerMap: removing container" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" Oct 30 22:26:22.626884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626785 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" Oct 30 22:26:22.626884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626810 4335 cpu_manager.go:414] "containerMap: removing container" podUID="b1655cc1-72e4-4f89-a69f-b351d3228a1c" containerName="revoke-keys" Oct 30 22:26:22.626884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626817 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1655cc1-72e4-4f89-a69f-b351d3228a1c" containerName="revoke-keys" Oct 30 22:26:22.626884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626831 4335 cpu_manager.go:414] "containerMap: removing container" podUID="60a004f7-2dc7-4b0e-a844-48404c0e35f2" containerName="istio-proxy" Oct 30 22:26:22.626884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626837 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a004f7-2dc7-4b0e-a844-48404c0e35f2" containerName="istio-proxy" Oct 30 22:26:22.626884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626847 4335 cpu_manager.go:414] "containerMap: removing container" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" Oct 30 22:26:22.626884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626854 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" Oct 30 22:26:22.626884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626869 4335 cpu_manager.go:414] "containerMap: removing container" podUID="0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" containerName="istio-proxy" Oct 30 22:26:22.626884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626876 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" containerName="istio-proxy" Oct 30 22:26:22.627198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626889 4335 cpu_manager.go:414] "containerMap: removing container" podUID="97d854e6-f96e-4c8f-906a-9f2a46eabc35" containerName="revoke-keys" Oct 30 22:26:22.627198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626896 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97d854e6-f96e-4c8f-906a-9f2a46eabc35" containerName="revoke-keys" Oct 30 22:26:22.627198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626934 4335 cpu_manager.go:414] "containerMap: removing container" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" Oct 30 22:26:22.627198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626954 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" Oct 30 22:26:22.627198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626966 4335 cpu_manager.go:414] "containerMap: removing container" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" Oct 30 22:26:22.627198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.626973 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" Oct 30 22:26:22.627198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.627105 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eefc8ca-80bf-4757-91ff-ef5d8ebffa23" containerName="main" Oct 30 22:26:22.627198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.627118 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03a0c73-ecb0-4b79-a8bc-3feaeebe16c5" containerName="main" Oct 30 22:26:22.627198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.627126 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b00cf8d-cd32-402b-8815-0c239365c8da" containerName="payload-pre-processing" Oct 30 22:26:22.627198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.627136 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" containerName="istio-proxy" Oct 30 22:26:22.627198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.627144 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="c592bee9-4da3-43f3-9347-a8ee5dc1fce6" containerName="main" Oct 30 22:26:22.627198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.627156 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1655cc1-72e4-4f89-a69f-b351d3228a1c" containerName="revoke-keys" Oct 30 22:26:22.627198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.627164 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a004f7-2dc7-4b0e-a844-48404c0e35f2" containerName="istio-proxy" Oct 30 22:26:22.627198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.627172 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4bee3da-5d5b-4663-9f31-bf0b2388e06e" containerName="main" Oct 30 22:26:22.627198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.627183 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="97d854e6-f96e-4c8f-906a-9f2a46eabc35" containerName="revoke-keys" Oct 30 22:26:22.631341 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.631318 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq" Oct 30 22:26:22.685674 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.685643 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq"] Oct 30 22:26:22.724299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.724269 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81201e64-6a4c-40d5-9d17-3b4cf575abd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-67657a-j7sbq\" (UID: \"81201e64-6a4c-40d5-9d17-3b4cf575abd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq" Oct 30 22:26:22.724496 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.724476 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81201e64-6a4c-40d5-9d17-3b4cf575abd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-67657a-j7sbq\" (UID: \"81201e64-6a4c-40d5-9d17-3b4cf575abd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq" Oct 30 22:26:22.724609 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.724541 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4gb5\" (UniqueName: \"kubernetes.io/projected/81201e64-6a4c-40d5-9d17-3b4cf575abd3-kube-api-access-v4gb5\") pod \"maas-api-revoke-keys-e2e-api-b-67657a-j7sbq\" (UID: \"81201e64-6a4c-40d5-9d17-3b4cf575abd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq" Oct 30 22:26:22.729434 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.729410 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:26:22.729539 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.729507 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:26:22.729658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.729607 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:26:22.733397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.733377 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:26:22.826018 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.825988 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81201e64-6a4c-40d5-9d17-3b4cf575abd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-67657a-j7sbq\" (UID: \"81201e64-6a4c-40d5-9d17-3b4cf575abd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq" Oct 30 22:26:22.826389 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.826128 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81201e64-6a4c-40d5-9d17-3b4cf575abd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-67657a-j7sbq\" (UID: \"81201e64-6a4c-40d5-9d17-3b4cf575abd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq" Oct 30 22:26:22.826389 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.826181 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4gb5\" (UniqueName: \"kubernetes.io/projected/81201e64-6a4c-40d5-9d17-3b4cf575abd3-kube-api-access-v4gb5\") pod \"maas-api-revoke-keys-e2e-api-b-67657a-j7sbq\" (UID: \"81201e64-6a4c-40d5-9d17-3b4cf575abd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq" Oct 30 22:26:22.826945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.826918 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81201e64-6a4c-40d5-9d17-3b4cf575abd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-67657a-j7sbq\" (UID: \"81201e64-6a4c-40d5-9d17-3b4cf575abd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq" Oct 30 22:26:22.829114 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.829088 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81201e64-6a4c-40d5-9d17-3b4cf575abd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-67657a-j7sbq\" (UID: \"81201e64-6a4c-40d5-9d17-3b4cf575abd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq" Oct 30 22:26:22.894808 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.894713 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4gb5\" (UniqueName: \"kubernetes.io/projected/81201e64-6a4c-40d5-9d17-3b4cf575abd3-kube-api-access-v4gb5\") pod \"maas-api-revoke-keys-e2e-api-b-67657a-j7sbq\" (UID: \"81201e64-6a4c-40d5-9d17-3b4cf575abd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq" Oct 30 22:26:22.948508 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:22.948478 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq" Oct 30 22:26:23.340546 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:23.340510 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq"] Oct 30 22:26:23.340711 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:26:23.340600 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81201e64_6a4c_40d5_9d17_3b4cf575abd3.slice/crio-494077f3162999b8a7623eb1af7bf2ce4671ebe74b7d15a09192bdccfefc9ff2 WatchSource:0}: Error finding container 494077f3162999b8a7623eb1af7bf2ce4671ebe74b7d15a09192bdccfefc9ff2: Status 404 returned error can't find the container with id 494077f3162999b8a7623eb1af7bf2ce4671ebe74b7d15a09192bdccfefc9ff2 Oct 30 22:26:23.826685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:23.826300 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq" event={"ID":"81201e64-6a4c-40d5-9d17-3b4cf575abd3","Type":"ContainerStarted","Data":"5996082feac6897b6ed100acdb21f72401dd3a33d774196c2a0c56272fbfb5ce"} Oct 30 22:26:23.826685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:23.826376 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq" event={"ID":"81201e64-6a4c-40d5-9d17-3b4cf575abd3","Type":"ContainerStarted","Data":"494077f3162999b8a7623eb1af7bf2ce4671ebe74b7d15a09192bdccfefc9ff2"} Oct 30 22:26:23.944440 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:23.944340 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq" podStartSLOduration=1.944323146 podStartE2EDuration="1.944323146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:23.941563716 +0000 UTC m=+2517.041995449" watchObservedRunningTime="2026-10-30 22:26:23.944323146 +0000 UTC m=+2517.044754878" Oct 30 22:26:24.600523 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:24.600475 4335 patch_prober.go:29] interesting pod/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:26:24.600730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:24.600542 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" podUID="58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:26:24.839984 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:24.839949 4335 generic.go:356] "Generic (PLEG): container finished" podID="81201e64-6a4c-40d5-9d17-3b4cf575abd3" containerID="5996082feac6897b6ed100acdb21f72401dd3a33d774196c2a0c56272fbfb5ce" exitCode=0 Oct 30 22:26:24.840368 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:24.840035 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq" event={"ID":"81201e64-6a4c-40d5-9d17-3b4cf575abd3","Type":"ContainerDied","Data":"5996082feac6897b6ed100acdb21f72401dd3a33d774196c2a0c56272fbfb5ce"} Oct 30 22:26:25.336884 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:26:25.336841 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:26:25.381246 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.381192 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:26:25.466166 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.466140 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-workload-certs\") pod \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " Oct 30 22:26:25.466302 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.466272 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istiod-ca-cert\") pod \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " Oct 30 22:26:25.466391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.466374 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-workload-socket\") pod \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " Oct 30 22:26:25.466434 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.466413 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-podinfo\") pod \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " Oct 30 22:26:25.466434 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.466420 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-workload-certs" pod "58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" (UID: "58adbdd2-9a58-48f3-bbc2-72c3ac297d5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:25.466498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.466447 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-token\" (UniqueName: \"kubernetes.io/projected/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-token\") pod \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " Oct 30 22:26:25.466498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.466476 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-data\") pod \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " Oct 30 22:26:25.466594 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.466530 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-credential-socket\") pod \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " Oct 30 22:26:25.466594 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.466564 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-kube-api-access-ntb87\" (UniqueName: \"kubernetes.io/projected/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-kube-api-access-ntb87\") pod \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " Oct 30 22:26:25.466718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.466595 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-envoy\") pod \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\" (UID: \"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a\") " Oct 30 22:26:25.466718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.466680 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istiod-ca-cert" pod "58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" (UID: "58adbdd2-9a58-48f3-bbc2-72c3ac297d5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:25.466987 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.466955 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-workload-socket" pod "58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" (UID: "58adbdd2-9a58-48f3-bbc2-72c3ac297d5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:25.467200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.466980 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-data" pod "58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" (UID: "58adbdd2-9a58-48f3-bbc2-72c3ac297d5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:25.467200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.467002 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-credential-socket" pod "58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" (UID: "58adbdd2-9a58-48f3-bbc2-72c3ac297d5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:25.467200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.467180 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:25.467200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.467204 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:25.467200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.467219 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:25.467200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.467233 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:25.467579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.467247 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:25.468848 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.468820 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-token" pod "58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" (UID: "58adbdd2-9a58-48f3-bbc2-72c3ac297d5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:25.468848 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.468829 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-podinfo" pod "58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" (UID: "58adbdd2-9a58-48f3-bbc2-72c3ac297d5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:25.469015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.468922 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-envoy" pod "58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" (UID: "58adbdd2-9a58-48f3-bbc2-72c3ac297d5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:25.469015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.468928 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-kube-api-access-ntb87" pod "58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" (UID: "58adbdd2-9a58-48f3-bbc2-72c3ac297d5a"). InnerVolumeSpecName "kube-api-access-ntb87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:25.568752 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.568721 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:25.568752 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.568745 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:25.568752 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.568755 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntb87\" (UniqueName: \"kubernetes.io/projected/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-kube-api-access-ntb87\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:25.568984 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.568764 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:25.852848 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.852809 4335 generic.go:356] "Generic (PLEG): container finished" podID="58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" containerID="9b99750cf13046cc3352d003c8e66342d562e720518a4e3f378c931820fec202" exitCode=0 Oct 30 22:26:25.853323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.852858 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" Oct 30 22:26:25.853323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.852903 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" event={"ID":"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a","Type":"ContainerDied","Data":"9b99750cf13046cc3352d003c8e66342d562e720518a4e3f378c931820fec202"} Oct 30 22:26:25.853323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.852972 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6" event={"ID":"58adbdd2-9a58-48f3-bbc2-72c3ac297d5a","Type":"ContainerDied","Data":"59cd99a47e532ed206dd102c8c91e813915dba4867c438a3a6348c1971f8bfea"} Oct 30 22:26:25.853323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.853011 4335 scope.go:122] "RemoveContainer" containerID="9b99750cf13046cc3352d003c8e66342d562e720518a4e3f378c931820fec202" Oct 30 22:26:25.870845 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.870825 4335 scope.go:122] "RemoveContainer" containerID="fb0c89f770165718e486203b6306cfe225cf43f9c91266f30e9e15e5dfbbe74f" Oct 30 22:26:25.885835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.885815 4335 scope.go:122] "RemoveContainer" containerID="9b99750cf13046cc3352d003c8e66342d562e720518a4e3f378c931820fec202" Oct 30 22:26:25.886179 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:26:25.886153 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b99750cf13046cc3352d003c8e66342d562e720518a4e3f378c931820fec202\": container with ID starting with 9b99750cf13046cc3352d003c8e66342d562e720518a4e3f378c931820fec202 not found: ID does not exist" containerID="9b99750cf13046cc3352d003c8e66342d562e720518a4e3f378c931820fec202" Oct 30 22:26:25.886244 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.886192 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b99750cf13046cc3352d003c8e66342d562e720518a4e3f378c931820fec202"} err="failed to get container status \"9b99750cf13046cc3352d003c8e66342d562e720518a4e3f378c931820fec202\": rpc error: code = NotFound desc = could not find container \"9b99750cf13046cc3352d003c8e66342d562e720518a4e3f378c931820fec202\": container with ID starting with 9b99750cf13046cc3352d003c8e66342d562e720518a4e3f378c931820fec202 not found: ID does not exist" Oct 30 22:26:25.886244 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.886220 4335 scope.go:122] "RemoveContainer" containerID="fb0c89f770165718e486203b6306cfe225cf43f9c91266f30e9e15e5dfbbe74f" Oct 30 22:26:25.886609 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:26:25.886578 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0c89f770165718e486203b6306cfe225cf43f9c91266f30e9e15e5dfbbe74f\": container with ID starting with fb0c89f770165718e486203b6306cfe225cf43f9c91266f30e9e15e5dfbbe74f not found: ID does not exist" containerID="fb0c89f770165718e486203b6306cfe225cf43f9c91266f30e9e15e5dfbbe74f" Oct 30 22:26:25.886787 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:25.886606 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0c89f770165718e486203b6306cfe225cf43f9c91266f30e9e15e5dfbbe74f"} err="failed to get container status \"fb0c89f770165718e486203b6306cfe225cf43f9c91266f30e9e15e5dfbbe74f\": rpc error: code = NotFound desc = could not find container \"fb0c89f770165718e486203b6306cfe225cf43f9c91266f30e9e15e5dfbbe74f\": container with ID starting with fb0c89f770165718e486203b6306cfe225cf43f9c91266f30e9e15e5dfbbe74f not found: ID does not exist" Oct 30 22:26:26.053281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:26.053186 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6"] Oct 30 22:26:26.068021 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:26.067992 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq" Oct 30 22:26:26.099939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:26.099905 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-25237a-openshift-default-84b9547bdf-gz5z6"] Oct 30 22:26:26.176122 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:26.176097 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81201e64-6a4c-40d5-9d17-3b4cf575abd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81201e64-6a4c-40d5-9d17-3b4cf575abd3-maas-api-xks-ca\") pod \"81201e64-6a4c-40d5-9d17-3b4cf575abd3\" (UID: \"81201e64-6a4c-40d5-9d17-3b4cf575abd3\") " Oct 30 22:26:26.176300 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:26.176189 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81201e64-6a4c-40d5-9d17-3b4cf575abd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81201e64-6a4c-40d5-9d17-3b4cf575abd3-maas-api-openshift-ca\") pod \"81201e64-6a4c-40d5-9d17-3b4cf575abd3\" (UID: \"81201e64-6a4c-40d5-9d17-3b4cf575abd3\") " Oct 30 22:26:26.176385 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:26.176341 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81201e64-6a4c-40d5-9d17-3b4cf575abd3-kube-api-access-v4gb5\" (UniqueName: \"kubernetes.io/projected/81201e64-6a4c-40d5-9d17-3b4cf575abd3-kube-api-access-v4gb5\") pod \"81201e64-6a4c-40d5-9d17-3b4cf575abd3\" (UID: \"81201e64-6a4c-40d5-9d17-3b4cf575abd3\") " Oct 30 22:26:26.176560 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:26.176527 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81201e64-6a4c-40d5-9d17-3b4cf575abd3-maas-api-openshift-ca" pod "81201e64-6a4c-40d5-9d17-3b4cf575abd3" (UID: "81201e64-6a4c-40d5-9d17-3b4cf575abd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:26.177520 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:26.177491 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81201e64-6a4c-40d5-9d17-3b4cf575abd3-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:26.178835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:26.178802 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81201e64-6a4c-40d5-9d17-3b4cf575abd3-maas-api-xks-ca" pod "81201e64-6a4c-40d5-9d17-3b4cf575abd3" (UID: "81201e64-6a4c-40d5-9d17-3b4cf575abd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:26.179055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:26.179033 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81201e64-6a4c-40d5-9d17-3b4cf575abd3-kube-api-access-v4gb5" pod "81201e64-6a4c-40d5-9d17-3b4cf575abd3" (UID: "81201e64-6a4c-40d5-9d17-3b4cf575abd3"). InnerVolumeSpecName "kube-api-access-v4gb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:26.279192 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:26.279153 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81201e64-6a4c-40d5-9d17-3b4cf575abd3-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:26.279192 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:26.279192 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4gb5\" (UniqueName: \"kubernetes.io/projected/81201e64-6a4c-40d5-9d17-3b4cf575abd3-kube-api-access-v4gb5\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:26.403977 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:26.403899 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:26:26.863661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:26.863635 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq" Oct 30 22:26:26.863661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:26.863642 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq" event={"ID":"81201e64-6a4c-40d5-9d17-3b4cf575abd3","Type":"ContainerDied","Data":"494077f3162999b8a7623eb1af7bf2ce4671ebe74b7d15a09192bdccfefc9ff2"} Oct 30 22:26:26.864018 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:26.863693 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="494077f3162999b8a7623eb1af7bf2ce4671ebe74b7d15a09192bdccfefc9ff2" Oct 30 22:26:27.058806 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.058778 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" path="/var/lib/kubelet/pods/58adbdd2-9a58-48f3-bbc2-72c3ac297d5a/volumes" Oct 30 22:26:27.069734 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.069680 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:26:27.187903 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.187880 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-data\") pod \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " Oct 30 22:26:27.188043 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.187910 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-credential-socket\") pod \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " Oct 30 22:26:27.188043 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.187929 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e3c2933-2586-4001-aa99-c18c1a6d44bb-kube-api-access-b7ldl\" (UniqueName: \"kubernetes.io/projected/5e3c2933-2586-4001-aa99-c18c1a6d44bb-kube-api-access-b7ldl\") pod \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " Oct 30 22:26:27.188043 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.187955 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istiod-ca-cert\") pod \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " Oct 30 22:26:27.188219 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.188066 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-token\" (UniqueName: \"kubernetes.io/projected/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-token\") pod \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " Oct 30 22:26:27.188219 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.188127 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-workload-socket\") pod \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " Oct 30 22:26:27.188219 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.188201 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-podinfo\") pod \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " Oct 30 22:26:27.188400 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.188222 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-credential-socket" pod "5e3c2933-2586-4001-aa99-c18c1a6d44bb" (UID: "5e3c2933-2586-4001-aa99-c18c1a6d44bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:27.188400 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.188196 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-data" pod "5e3c2933-2586-4001-aa99-c18c1a6d44bb" (UID: "5e3c2933-2586-4001-aa99-c18c1a6d44bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:27.188400 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.188290 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-workload-certs\") pod \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " Oct 30 22:26:27.188400 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.188332 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-envoy\") pod \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\" (UID: \"5e3c2933-2586-4001-aa99-c18c1a6d44bb\") " Oct 30 22:26:27.188400 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.188334 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istiod-ca-cert" pod "5e3c2933-2586-4001-aa99-c18c1a6d44bb" (UID: "5e3c2933-2586-4001-aa99-c18c1a6d44bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:27.188693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.188420 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-workload-socket" pod "5e3c2933-2586-4001-aa99-c18c1a6d44bb" (UID: "5e3c2933-2586-4001-aa99-c18c1a6d44bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:27.188693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.188546 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-workload-certs" pod "5e3c2933-2586-4001-aa99-c18c1a6d44bb" (UID: "5e3c2933-2586-4001-aa99-c18c1a6d44bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:27.188900 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.188882 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:27.188980 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.188905 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:27.188980 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.188920 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:27.188980 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.188934 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:27.189152 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.188991 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:27.190772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.190737 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-token" pod "5e3c2933-2586-4001-aa99-c18c1a6d44bb" (UID: "5e3c2933-2586-4001-aa99-c18c1a6d44bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:27.191417 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.191392 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-podinfo" pod "5e3c2933-2586-4001-aa99-c18c1a6d44bb" (UID: "5e3c2933-2586-4001-aa99-c18c1a6d44bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:27.191806 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.191766 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-envoy" pod "5e3c2933-2586-4001-aa99-c18c1a6d44bb" (UID: "5e3c2933-2586-4001-aa99-c18c1a6d44bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:27.194847 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.194820 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3c2933-2586-4001-aa99-c18c1a6d44bb-kube-api-access-b7ldl" pod "5e3c2933-2586-4001-aa99-c18c1a6d44bb" (UID: "5e3c2933-2586-4001-aa99-c18c1a6d44bb"). InnerVolumeSpecName "kube-api-access-b7ldl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:27.289800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.289771 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:27.289800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.289797 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:27.289989 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.289807 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7ldl\" (UniqueName: \"kubernetes.io/projected/5e3c2933-2586-4001-aa99-c18c1a6d44bb-kube-api-access-b7ldl\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:27.289989 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.289815 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e3c2933-2586-4001-aa99-c18c1a6d44bb-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:27.878484 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.878446 4335 generic.go:356] "Generic (PLEG): container finished" podID="5e3c2933-2586-4001-aa99-c18c1a6d44bb" containerID="11fce6f669b7c04c675f8f922eb71fcc58e0adfab32b23bbaa5fa495eebfa8f1" exitCode=0 Oct 30 22:26:27.879001 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.878517 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" Oct 30 22:26:27.879001 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.878532 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" event={"ID":"5e3c2933-2586-4001-aa99-c18c1a6d44bb","Type":"ContainerDied","Data":"11fce6f669b7c04c675f8f922eb71fcc58e0adfab32b23bbaa5fa495eebfa8f1"} Oct 30 22:26:27.879001 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.878601 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn" event={"ID":"5e3c2933-2586-4001-aa99-c18c1a6d44bb","Type":"ContainerDied","Data":"c8b77cc4664dd7a03ca4b4dd52b2246ee3fb24f5d40f5ab7a106359e6a11cd15"} Oct 30 22:26:27.879001 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.878644 4335 scope.go:122] "RemoveContainer" containerID="11fce6f669b7c04c675f8f922eb71fcc58e0adfab32b23bbaa5fa495eebfa8f1" Oct 30 22:26:27.894114 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.894092 4335 scope.go:122] "RemoveContainer" containerID="868c00326a18b5e45b5760dde9dfd0fb89c3bf7d9aff85cecde3eb47a5b3350a" Oct 30 22:26:27.904211 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.904190 4335 scope.go:122] "RemoveContainer" containerID="11fce6f669b7c04c675f8f922eb71fcc58e0adfab32b23bbaa5fa495eebfa8f1" Oct 30 22:26:27.904549 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:26:27.904509 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11fce6f669b7c04c675f8f922eb71fcc58e0adfab32b23bbaa5fa495eebfa8f1\": container with ID starting with 11fce6f669b7c04c675f8f922eb71fcc58e0adfab32b23bbaa5fa495eebfa8f1 not found: ID does not exist" containerID="11fce6f669b7c04c675f8f922eb71fcc58e0adfab32b23bbaa5fa495eebfa8f1" Oct 30 22:26:27.904678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.904561 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11fce6f669b7c04c675f8f922eb71fcc58e0adfab32b23bbaa5fa495eebfa8f1"} err="failed to get container status \"11fce6f669b7c04c675f8f922eb71fcc58e0adfab32b23bbaa5fa495eebfa8f1\": rpc error: code = NotFound desc = could not find container \"11fce6f669b7c04c675f8f922eb71fcc58e0adfab32b23bbaa5fa495eebfa8f1\": container with ID starting with 11fce6f669b7c04c675f8f922eb71fcc58e0adfab32b23bbaa5fa495eebfa8f1 not found: ID does not exist" Oct 30 22:26:27.904678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.904587 4335 scope.go:122] "RemoveContainer" containerID="868c00326a18b5e45b5760dde9dfd0fb89c3bf7d9aff85cecde3eb47a5b3350a" Oct 30 22:26:27.904922 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:26:27.904899 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868c00326a18b5e45b5760dde9dfd0fb89c3bf7d9aff85cecde3eb47a5b3350a\": container with ID starting with 868c00326a18b5e45b5760dde9dfd0fb89c3bf7d9aff85cecde3eb47a5b3350a not found: ID does not exist" containerID="868c00326a18b5e45b5760dde9dfd0fb89c3bf7d9aff85cecde3eb47a5b3350a" Oct 30 22:26:27.904973 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:27.904934 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868c00326a18b5e45b5760dde9dfd0fb89c3bf7d9aff85cecde3eb47a5b3350a"} err="failed to get container status \"868c00326a18b5e45b5760dde9dfd0fb89c3bf7d9aff85cecde3eb47a5b3350a\": rpc error: code = NotFound desc = could not find container \"868c00326a18b5e45b5760dde9dfd0fb89c3bf7d9aff85cecde3eb47a5b3350a\": container with ID starting with 868c00326a18b5e45b5760dde9dfd0fb89c3bf7d9aff85cecde3eb47a5b3350a not found: ID does not exist" Oct 30 22:26:28.094120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:28.094090 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn"] Oct 30 22:26:28.146070 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:28.146000 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a1024a-openshift-default-5c8c69c7bc-hf7cn"] Oct 30 22:26:29.058470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:29.058437 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e3c2933-2586-4001-aa99-c18c1a6d44bb" path="/var/lib/kubelet/pods/5e3c2933-2586-4001-aa99-c18c1a6d44bb/volumes" Oct 30 22:26:29.752496 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:29.752459 4335 scope.go:122] "RemoveContainer" containerID="6cd0eb93248c093fa5c408623500ecdaca159c9dad58bfcae72782c5a763fb49" Oct 30 22:26:32.775081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.775042 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c"] Oct 30 22:26:32.775806 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.775785 4335 cpu_manager.go:414] "containerMap: removing container" podUID="58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" containerName="istio-proxy" Oct 30 22:26:32.775806 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.775807 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" containerName="istio-proxy" Oct 30 22:26:32.776065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.775827 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5e3c2933-2586-4001-aa99-c18c1a6d44bb" containerName="istio-proxy" Oct 30 22:26:32.776065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.775834 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e3c2933-2586-4001-aa99-c18c1a6d44bb" containerName="istio-proxy" Oct 30 22:26:32.776065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.775871 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5e3c2933-2586-4001-aa99-c18c1a6d44bb" containerName="istio-proxy" Oct 30 22:26:32.776065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.775880 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e3c2933-2586-4001-aa99-c18c1a6d44bb" containerName="istio-proxy" Oct 30 22:26:32.776065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.775890 4335 cpu_manager.go:414] "containerMap: removing container" podUID="58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" containerName="istio-proxy" Oct 30 22:26:32.776065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.775896 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" containerName="istio-proxy" Oct 30 22:26:32.776065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.775929 4335 cpu_manager.go:414] "containerMap: removing container" podUID="81201e64-6a4c-40d5-9d17-3b4cf575abd3" containerName="revoke-keys" Oct 30 22:26:32.776065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.775937 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81201e64-6a4c-40d5-9d17-3b4cf575abd3" containerName="revoke-keys" Oct 30 22:26:32.776452 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.776074 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" containerName="istio-proxy" Oct 30 22:26:32.776452 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.776089 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e3c2933-2586-4001-aa99-c18c1a6d44bb" containerName="istio-proxy" Oct 30 22:26:32.776452 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.776102 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f7b76ed-f54c-4ff1-930a-c5f183bc72ec" containerName="istio-proxy" Oct 30 22:26:32.776452 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.776115 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e3c2933-2586-4001-aa99-c18c1a6d44bb" containerName="istio-proxy" Oct 30 22:26:32.776452 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.776123 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="58adbdd2-9a58-48f3-bbc2-72c3ac297d5a" containerName="istio-proxy" Oct 30 22:26:32.776452 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.776138 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="81201e64-6a4c-40d5-9d17-3b4cf575abd3" containerName="revoke-keys" Oct 30 22:26:32.781861 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.781838 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:32.835652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.833689 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-87b6b3-openshift-default-dockercfg-cs98q\"" Oct 30 22:26:32.943807 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.943726 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c"] Oct 30 22:26:32.966926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.966896 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-workload-socket\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:32.966926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.966936 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-podinfo\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:32.967240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.966960 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-token\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:32.967240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.967000 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istiod-ca-cert\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:32.967240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.967030 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-envoy\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:32.967240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.967053 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9w5g\" (UniqueName: \"kubernetes.io/projected/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-kube-api-access-c9w5g\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:32.967240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.967109 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-workload-certs\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:32.967240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.967150 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-data\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:32.967240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:32.967172 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-credential-socket\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.030170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.030095 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56"] Oct 30 22:26:33.068522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.068489 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-workload-certs\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.068716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.068565 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-data\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.068716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.068599 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-credential-socket\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.068716 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.068698 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-workload-socket\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.068838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.068736 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-podinfo\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.068838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.068767 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-token\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.068838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.068826 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istiod-ca-cert\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.068934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.068842 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-workload-certs\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.068934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.068870 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-envoy\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.068934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.068902 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9w5g\" (UniqueName: \"kubernetes.io/projected/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-kube-api-access-c9w5g\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.069033 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.069021 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-workload-socket\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.069191 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.069171 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-data\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.069819 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.069359 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-credential-socket\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.069819 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.069588 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istiod-ca-cert\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.071667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.071645 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-envoy\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.074143 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.074120 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-podinfo\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.129694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.129667 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-token\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.129834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.129742 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9w5g\" (UniqueName: \"kubernetes.io/projected/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-kube-api-access-c9w5g\") pod \"e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.138451 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.138427 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx"] Oct 30 22:26:33.427291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.421468 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:33.924566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.924536 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c"] Oct 30 22:26:33.961158 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:26:33.961126 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ad9d54_24dc_40ea_9397_5bfc51b9d786.slice/crio-833b66cbd97147912ab34d3a90bae71c4bd1d9ccc4ac6a043be789bc70f7a904 WatchSource:0}: Error finding container 833b66cbd97147912ab34d3a90bae71c4bd1d9ccc4ac6a043be789bc70f7a904: Status 404 returned error can't find the container with id 833b66cbd97147912ab34d3a90bae71c4bd1d9ccc4ac6a043be789bc70f7a904 Oct 30 22:26:33.963554 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.963518 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:33.963691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.963589 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:33.963691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.963639 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:33.982407 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:33.982380 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" event={"ID":"f2ad9d54-24dc-40ea-9397-5bfc51b9d786","Type":"ContainerStarted","Data":"833b66cbd97147912ab34d3a90bae71c4bd1d9ccc4ac6a043be789bc70f7a904"} Oct 30 22:26:34.574046 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.565164 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2"] Oct 30 22:26:34.581414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.581383 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2" Oct 30 22:26:34.610297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.610266 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb"] Oct 30 22:26:34.615439 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.615411 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb" Oct 30 22:26:34.615745 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.615724 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2"] Oct 30 22:26:34.634141 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.634116 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:26:34.634252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.634163 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:26:34.634252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.634201 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:26:34.634346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.634253 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:26:34.674166 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.674137 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb"] Oct 30 22:26:34.700608 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.700575 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97439aa6-0ced-4c9b-bd2e-73039917e317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b61b44bd-g5hbb\" (UID: \"97439aa6-0ced-4c9b-bd2e-73039917e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb" Oct 30 22:26:34.700780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.700707 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zhq4\" (UniqueName: \"kubernetes.io/projected/97439aa6-0ced-4c9b-bd2e-73039917e317-kube-api-access-9zhq4\") pod \"maas-api-revoke-sub-b61b44bd-g5hbb\" (UID: \"97439aa6-0ced-4c9b-bd2e-73039917e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb" Oct 30 22:26:34.700780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.700755 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vf54\" (UniqueName: \"kubernetes.io/projected/2f9df26f-a88b-431d-b6dc-d608694b1b55-kube-api-access-2vf54\") pod \"maas-api-revoke-sub-9d058665-w2st2\" (UID: \"2f9df26f-a88b-431d-b6dc-d608694b1b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2" Oct 30 22:26:34.700900 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.700841 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97439aa6-0ced-4c9b-bd2e-73039917e317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b61b44bd-g5hbb\" (UID: \"97439aa6-0ced-4c9b-bd2e-73039917e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb" Oct 30 22:26:34.700900 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.700874 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9df26f-a88b-431d-b6dc-d608694b1b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d058665-w2st2\" (UID: \"2f9df26f-a88b-431d-b6dc-d608694b1b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2" Oct 30 22:26:34.701052 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.701030 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9df26f-a88b-431d-b6dc-d608694b1b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d058665-w2st2\" (UID: \"2f9df26f-a88b-431d-b6dc-d608694b1b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2" Oct 30 22:26:34.802257 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.802224 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97439aa6-0ced-4c9b-bd2e-73039917e317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b61b44bd-g5hbb\" (UID: \"97439aa6-0ced-4c9b-bd2e-73039917e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb" Oct 30 22:26:34.802466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.802287 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9df26f-a88b-431d-b6dc-d608694b1b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d058665-w2st2\" (UID: \"2f9df26f-a88b-431d-b6dc-d608694b1b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2" Oct 30 22:26:34.802466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.802380 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9df26f-a88b-431d-b6dc-d608694b1b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d058665-w2st2\" (UID: \"2f9df26f-a88b-431d-b6dc-d608694b1b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2" Oct 30 22:26:34.802466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.802416 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97439aa6-0ced-4c9b-bd2e-73039917e317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b61b44bd-g5hbb\" (UID: \"97439aa6-0ced-4c9b-bd2e-73039917e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb" Oct 30 22:26:34.802466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.802460 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zhq4\" (UniqueName: \"kubernetes.io/projected/97439aa6-0ced-4c9b-bd2e-73039917e317-kube-api-access-9zhq4\") pod \"maas-api-revoke-sub-b61b44bd-g5hbb\" (UID: \"97439aa6-0ced-4c9b-bd2e-73039917e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb" Oct 30 22:26:34.802746 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.802513 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vf54\" (UniqueName: \"kubernetes.io/projected/2f9df26f-a88b-431d-b6dc-d608694b1b55-kube-api-access-2vf54\") pod \"maas-api-revoke-sub-9d058665-w2st2\" (UID: \"2f9df26f-a88b-431d-b6dc-d608694b1b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2" Oct 30 22:26:34.804995 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.804962 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9df26f-a88b-431d-b6dc-d608694b1b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d058665-w2st2\" (UID: \"2f9df26f-a88b-431d-b6dc-d608694b1b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2" Oct 30 22:26:34.805806 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.805773 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97439aa6-0ced-4c9b-bd2e-73039917e317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b61b44bd-g5hbb\" (UID: \"97439aa6-0ced-4c9b-bd2e-73039917e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb" Oct 30 22:26:34.807073 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.807016 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9df26f-a88b-431d-b6dc-d608694b1b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d058665-w2st2\" (UID: \"2f9df26f-a88b-431d-b6dc-d608694b1b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2" Oct 30 22:26:34.815172 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.815146 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97439aa6-0ced-4c9b-bd2e-73039917e317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b61b44bd-g5hbb\" (UID: \"97439aa6-0ced-4c9b-bd2e-73039917e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb" Oct 30 22:26:34.863949 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.863882 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vf54\" (UniqueName: \"kubernetes.io/projected/2f9df26f-a88b-431d-b6dc-d608694b1b55-kube-api-access-2vf54\") pod \"maas-api-revoke-sub-9d058665-w2st2\" (UID: \"2f9df26f-a88b-431d-b6dc-d608694b1b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2" Oct 30 22:26:34.865239 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.865213 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zhq4\" (UniqueName: \"kubernetes.io/projected/97439aa6-0ced-4c9b-bd2e-73039917e317-kube-api-access-9zhq4\") pod \"maas-api-revoke-sub-b61b44bd-g5hbb\" (UID: \"97439aa6-0ced-4c9b-bd2e-73039917e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb" Oct 30 22:26:34.901891 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.901862 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2" Oct 30 22:26:34.951541 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.950844 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb" Oct 30 22:26:34.999667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:34.996298 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" event={"ID":"f2ad9d54-24dc-40ea-9397-5bfc51b9d786","Type":"ContainerStarted","Data":"8cba0bc44621f7c2cec491e32470b93bc6487a4773f2cf7f0fd490b4ead19284"} Oct 30 22:26:35.120170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:35.120073 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" podStartSLOduration=3.120053282 podStartE2EDuration="3.120053282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:35.11621924 +0000 UTC m=+2528.216650973" watchObservedRunningTime="2026-10-30 22:26:35.120053282 +0000 UTC m=+2528.220485015" Oct 30 22:26:35.345389 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:26:35.345353 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9df26f_a88b_431d_b6dc_d608694b1b55.slice/crio-f446331bd1d22146c09d19cfcf1409356131cf45d3ef51ada2c61a90d0dd0d19 WatchSource:0}: Error finding container f446331bd1d22146c09d19cfcf1409356131cf45d3ef51ada2c61a90d0dd0d19: Status 404 returned error can't find the container with id f446331bd1d22146c09d19cfcf1409356131cf45d3ef51ada2c61a90d0dd0d19 Oct 30 22:26:35.346451 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:35.346426 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2"] Oct 30 22:26:35.402153 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:35.402126 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb"] Oct 30 22:26:35.425607 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:35.425576 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:35.432437 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:35.432377 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:35.498812 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:26:35.498779 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97439aa6_0ced_4c9b_bd2e_73039917e317.slice/crio-7507ec007258331b6205882a5fbf2571c41f72362e6d2e712c0a7850112275ba WatchSource:0}: Error finding container 7507ec007258331b6205882a5fbf2571c41f72362e6d2e712c0a7850112275ba: Status 404 returned error can't find the container with id 7507ec007258331b6205882a5fbf2571c41f72362e6d2e712c0a7850112275ba Oct 30 22:26:36.016005 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:36.015910 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2" event={"ID":"2f9df26f-a88b-431d-b6dc-d608694b1b55","Type":"ContainerStarted","Data":"3f21a62b396225d2e85ff508ba16d64d2cb433708e904565d233c4f28292fe3b"} Oct 30 22:26:36.016005 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:36.015979 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2" event={"ID":"2f9df26f-a88b-431d-b6dc-d608694b1b55","Type":"ContainerStarted","Data":"f446331bd1d22146c09d19cfcf1409356131cf45d3ef51ada2c61a90d0dd0d19"} Oct 30 22:26:36.021522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:36.020867 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb" event={"ID":"97439aa6-0ced-4c9b-bd2e-73039917e317","Type":"ContainerStarted","Data":"e7b576c666b5920a28a40d688053f4abd2d847cde69dc86985ec23ca585a1087"} Oct 30 22:26:36.021522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:36.020936 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb" event={"ID":"97439aa6-0ced-4c9b-bd2e-73039917e317","Type":"ContainerStarted","Data":"7507ec007258331b6205882a5fbf2571c41f72362e6d2e712c0a7850112275ba"} Oct 30 22:26:36.021522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:36.021371 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:36.022282 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:36.022259 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:26:36.134312 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:36.134244 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2" podStartSLOduration=2.13422472 podStartE2EDuration="2.13422472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:36.131860607 +0000 UTC m=+2529.232292340" watchObservedRunningTime="2026-10-30 22:26:36.13422472 +0000 UTC m=+2529.234656457" Oct 30 22:26:36.370626 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:36.370556 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb" podStartSLOduration=2.370541806 podStartE2EDuration="2.370541806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:36.367767366 +0000 UTC m=+2529.468199100" watchObservedRunningTime="2026-10-30 22:26:36.370541806 +0000 UTC m=+2529.470973538" Oct 30 22:26:37.046681 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:37.046637 4335 generic.go:356] "Generic (PLEG): container finished" podID="97439aa6-0ced-4c9b-bd2e-73039917e317" containerID="e7b576c666b5920a28a40d688053f4abd2d847cde69dc86985ec23ca585a1087" exitCode=0 Oct 30 22:26:37.047171 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:37.046728 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb" event={"ID":"97439aa6-0ced-4c9b-bd2e-73039917e317","Type":"ContainerDied","Data":"e7b576c666b5920a28a40d688053f4abd2d847cde69dc86985ec23ca585a1087"} Oct 30 22:26:37.048695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:37.048658 4335 generic.go:356] "Generic (PLEG): container finished" podID="2f9df26f-a88b-431d-b6dc-d608694b1b55" containerID="3f21a62b396225d2e85ff508ba16d64d2cb433708e904565d233c4f28292fe3b" exitCode=0 Oct 30 22:26:37.048837 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:37.048733 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2" event={"ID":"2f9df26f-a88b-431d-b6dc-d608694b1b55","Type":"ContainerDied","Data":"3f21a62b396225d2e85ff508ba16d64d2cb433708e904565d233c4f28292fe3b"} Oct 30 22:26:38.030742 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.030655 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56" podUID="d2c9a56d-93df-4a25-bec4-6fff59fac92a" containerName="payload-processing" containerID="cri-o://c71c6953b58a3b4625dd3f672eb249693875c14efcd5a497a719574a2246a442" gracePeriod=25 Oct 30 22:26:38.057026 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.057002 4335 generic.go:356] "Generic (PLEG): container finished" podID="1657e3aa-0cfe-49e5-b8ce-21888a334461" containerID="f305f04e50ee079295e625005f01f05fef22490b4cc3c54a926f557a040df7b4" exitCode=0 Oct 30 22:26:38.057412 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.057084 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" event={"ID":"1657e3aa-0cfe-49e5-b8ce-21888a334461","Type":"ContainerDied","Data":"f305f04e50ee079295e625005f01f05fef22490b4cc3c54a926f557a040df7b4"} Oct 30 22:26:38.057493 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.057423 4335 scope.go:122] "RemoveContainer" containerID="f305f04e50ee079295e625005f01f05fef22490b4cc3c54a926f557a040df7b4" Oct 30 22:26:38.059438 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.059341 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:38.061454 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.061427 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:38.061583 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.061482 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:38.146788 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.146701 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx" podUID="0b3173ec-26e6-4be5-9d48-911da303107b" containerName="payload-pre-processing" containerID="cri-o://fb2bd9c0e515c45ccf7e4509858e4145c41f720232fd2bdc8c2869109a88c8be" gracePeriod=25 Oct 30 22:26:38.275208 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.275181 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb" Oct 30 22:26:38.280768 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.280489 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2" Oct 30 22:26:38.357803 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.350678 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97439aa6-0ced-4c9b-bd2e-73039917e317-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97439aa6-0ced-4c9b-bd2e-73039917e317-maas-api-xks-ca\") pod \"97439aa6-0ced-4c9b-bd2e-73039917e317\" (UID: \"97439aa6-0ced-4c9b-bd2e-73039917e317\") " Oct 30 22:26:38.357803 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.351312 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f9df26f-a88b-431d-b6dc-d608694b1b55-kube-api-access-2vf54\" (UniqueName: \"kubernetes.io/projected/2f9df26f-a88b-431d-b6dc-d608694b1b55-kube-api-access-2vf54\") pod \"2f9df26f-a88b-431d-b6dc-d608694b1b55\" (UID: \"2f9df26f-a88b-431d-b6dc-d608694b1b55\") " Oct 30 22:26:38.357803 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.351448 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97439aa6-0ced-4c9b-bd2e-73039917e317-kube-api-access-9zhq4\" (UniqueName: \"kubernetes.io/projected/97439aa6-0ced-4c9b-bd2e-73039917e317-kube-api-access-9zhq4\") pod \"97439aa6-0ced-4c9b-bd2e-73039917e317\" (UID: \"97439aa6-0ced-4c9b-bd2e-73039917e317\") " Oct 30 22:26:38.357803 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.351525 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97439aa6-0ced-4c9b-bd2e-73039917e317-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97439aa6-0ced-4c9b-bd2e-73039917e317-maas-api-openshift-ca\") pod \"97439aa6-0ced-4c9b-bd2e-73039917e317\" (UID: \"97439aa6-0ced-4c9b-bd2e-73039917e317\") " Oct 30 22:26:38.357803 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.351551 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f9df26f-a88b-431d-b6dc-d608694b1b55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9df26f-a88b-431d-b6dc-d608694b1b55-maas-api-openshift-ca\") pod \"2f9df26f-a88b-431d-b6dc-d608694b1b55\" (UID: \"2f9df26f-a88b-431d-b6dc-d608694b1b55\") " Oct 30 22:26:38.357803 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.351606 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f9df26f-a88b-431d-b6dc-d608694b1b55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9df26f-a88b-431d-b6dc-d608694b1b55-maas-api-xks-ca\") pod \"2f9df26f-a88b-431d-b6dc-d608694b1b55\" (UID: \"2f9df26f-a88b-431d-b6dc-d608694b1b55\") " Oct 30 22:26:38.357803 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.352760 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97439aa6-0ced-4c9b-bd2e-73039917e317-maas-api-openshift-ca" pod "97439aa6-0ced-4c9b-bd2e-73039917e317" (UID: "97439aa6-0ced-4c9b-bd2e-73039917e317"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:38.357803 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.353605 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f9df26f-a88b-431d-b6dc-d608694b1b55-maas-api-openshift-ca" pod "2f9df26f-a88b-431d-b6dc-d608694b1b55" (UID: "2f9df26f-a88b-431d-b6dc-d608694b1b55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:38.357803 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.356103 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97439aa6-0ced-4c9b-bd2e-73039917e317-maas-api-xks-ca" pod "97439aa6-0ced-4c9b-bd2e-73039917e317" (UID: "97439aa6-0ced-4c9b-bd2e-73039917e317"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:38.358439 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.358361 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9df26f-a88b-431d-b6dc-d608694b1b55-kube-api-access-2vf54" pod "2f9df26f-a88b-431d-b6dc-d608694b1b55" (UID: "2f9df26f-a88b-431d-b6dc-d608694b1b55"). InnerVolumeSpecName "kube-api-access-2vf54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:38.362685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.359771 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f9df26f-a88b-431d-b6dc-d608694b1b55-maas-api-xks-ca" pod "2f9df26f-a88b-431d-b6dc-d608694b1b55" (UID: "2f9df26f-a88b-431d-b6dc-d608694b1b55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:38.362685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.359928 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97439aa6-0ced-4c9b-bd2e-73039917e317-kube-api-access-9zhq4" pod "97439aa6-0ced-4c9b-bd2e-73039917e317" (UID: "97439aa6-0ced-4c9b-bd2e-73039917e317"). InnerVolumeSpecName "kube-api-access-9zhq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:38.429653 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.418890 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56" Oct 30 22:26:38.456985 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.456951 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97439aa6-0ced-4c9b-bd2e-73039917e317-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:38.456985 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.456981 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vf54\" (UniqueName: \"kubernetes.io/projected/2f9df26f-a88b-431d-b6dc-d608694b1b55-kube-api-access-2vf54\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:38.457194 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.456997 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zhq4\" (UniqueName: \"kubernetes.io/projected/97439aa6-0ced-4c9b-bd2e-73039917e317-kube-api-access-9zhq4\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:38.457194 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.457010 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97439aa6-0ced-4c9b-bd2e-73039917e317-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:38.457194 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.457023 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9df26f-a88b-431d-b6dc-d608694b1b55-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:38.457194 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.457035 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9df26f-a88b-431d-b6dc-d608694b1b55-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:38.504814 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.504789 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx" Oct 30 22:26:38.558281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.558246 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2c9a56d-93df-4a25-bec4-6fff59fac92a-kube-api-access-wh7fv\" (UniqueName: \"kubernetes.io/projected/d2c9a56d-93df-4a25-bec4-6fff59fac92a-kube-api-access-wh7fv\") pod \"d2c9a56d-93df-4a25-bec4-6fff59fac92a\" (UID: \"d2c9a56d-93df-4a25-bec4-6fff59fac92a\") " Oct 30 22:26:38.558477 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.558309 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2c9a56d-93df-4a25-bec4-6fff59fac92a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2c9a56d-93df-4a25-bec4-6fff59fac92a-plugins-config-volume\") pod \"d2c9a56d-93df-4a25-bec4-6fff59fac92a\" (UID: \"d2c9a56d-93df-4a25-bec4-6fff59fac92a\") " Oct 30 22:26:38.558477 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.558343 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b3173ec-26e6-4be5-9d48-911da303107b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b3173ec-26e6-4be5-9d48-911da303107b-plugins-config-volume\") pod \"0b3173ec-26e6-4be5-9d48-911da303107b\" (UID: \"0b3173ec-26e6-4be5-9d48-911da303107b\") " Oct 30 22:26:38.558599 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.558546 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b3173ec-26e6-4be5-9d48-911da303107b-kube-api-access-82bkj\" (UniqueName: \"kubernetes.io/projected/0b3173ec-26e6-4be5-9d48-911da303107b-kube-api-access-82bkj\") pod \"0b3173ec-26e6-4be5-9d48-911da303107b\" (UID: \"0b3173ec-26e6-4be5-9d48-911da303107b\") " Oct 30 22:26:38.558748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.558714 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b3173ec-26e6-4be5-9d48-911da303107b-plugins-config-volume" pod "0b3173ec-26e6-4be5-9d48-911da303107b" (UID: "0b3173ec-26e6-4be5-9d48-911da303107b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:38.559149 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.559035 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b3173ec-26e6-4be5-9d48-911da303107b-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:38.559506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.559472 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c9a56d-93df-4a25-bec4-6fff59fac92a-plugins-config-volume" pod "d2c9a56d-93df-4a25-bec4-6fff59fac92a" (UID: "d2c9a56d-93df-4a25-bec4-6fff59fac92a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:38.560720 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.560685 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c9a56d-93df-4a25-bec4-6fff59fac92a-kube-api-access-wh7fv" pod "d2c9a56d-93df-4a25-bec4-6fff59fac92a" (UID: "d2c9a56d-93df-4a25-bec4-6fff59fac92a"). InnerVolumeSpecName "kube-api-access-wh7fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:38.561203 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.561174 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b3173ec-26e6-4be5-9d48-911da303107b-kube-api-access-82bkj" pod "0b3173ec-26e6-4be5-9d48-911da303107b" (UID: "0b3173ec-26e6-4be5-9d48-911da303107b"). InnerVolumeSpecName "kube-api-access-82bkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:38.659649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.659602 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82bkj\" (UniqueName: \"kubernetes.io/projected/0b3173ec-26e6-4be5-9d48-911da303107b-kube-api-access-82bkj\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:38.659649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.659650 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh7fv\" (UniqueName: \"kubernetes.io/projected/d2c9a56d-93df-4a25-bec4-6fff59fac92a-kube-api-access-wh7fv\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:38.659829 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:38.659663 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2c9a56d-93df-4a25-bec4-6fff59fac92a-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:39.065951 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.065918 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2" event={"ID":"2f9df26f-a88b-431d-b6dc-d608694b1b55","Type":"ContainerDied","Data":"f446331bd1d22146c09d19cfcf1409356131cf45d3ef51ada2c61a90d0dd0d19"} Oct 30 22:26:39.066336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.065962 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f446331bd1d22146c09d19cfcf1409356131cf45d3ef51ada2c61a90d0dd0d19" Oct 30 22:26:39.066336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.066014 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2" Oct 30 22:26:39.070318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.070290 4335 generic.go:356] "Generic (PLEG): container finished" podID="0b3173ec-26e6-4be5-9d48-911da303107b" containerID="fb2bd9c0e515c45ccf7e4509858e4145c41f720232fd2bdc8c2869109a88c8be" exitCode=0 Oct 30 22:26:39.070422 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.070342 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx" Oct 30 22:26:39.070422 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.070391 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx" event={"ID":"0b3173ec-26e6-4be5-9d48-911da303107b","Type":"ContainerDied","Data":"fb2bd9c0e515c45ccf7e4509858e4145c41f720232fd2bdc8c2869109a88c8be"} Oct 30 22:26:39.070540 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.070430 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx" event={"ID":"0b3173ec-26e6-4be5-9d48-911da303107b","Type":"ContainerDied","Data":"90d05355e912bde1bc7d372dffdbc6d8858498985badb37717a7158b7dee89bb"} Oct 30 22:26:39.070540 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.070455 4335 scope.go:122] "RemoveContainer" containerID="fb2bd9c0e515c45ccf7e4509858e4145c41f720232fd2bdc8c2869109a88c8be" Oct 30 22:26:39.073629 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.073587 4335 generic.go:356] "Generic (PLEG): container finished" podID="d2c9a56d-93df-4a25-bec4-6fff59fac92a" containerID="c71c6953b58a3b4625dd3f672eb249693875c14efcd5a497a719574a2246a442" exitCode=0 Oct 30 22:26:39.073740 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.073665 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56" Oct 30 22:26:39.074519 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.073959 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56" event={"ID":"d2c9a56d-93df-4a25-bec4-6fff59fac92a","Type":"ContainerDied","Data":"c71c6953b58a3b4625dd3f672eb249693875c14efcd5a497a719574a2246a442"} Oct 30 22:26:39.074519 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.074060 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56" event={"ID":"d2c9a56d-93df-4a25-bec4-6fff59fac92a","Type":"ContainerDied","Data":"8766c1cd819f1940a20a513d6c39de02e23975df6cfc565ec05dd68aa0a66ccd"} Oct 30 22:26:39.079549 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.079528 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" event={"ID":"1657e3aa-0cfe-49e5-b8ce-21888a334461","Type":"ContainerStarted","Data":"002618aed2cde0b287623624cd9bb75fd00ab8f7a9da4b7e07cc4ec5ac1d1d3e"} Oct 30 22:26:39.081884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.081864 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb" event={"ID":"97439aa6-0ced-4c9b-bd2e-73039917e317","Type":"ContainerDied","Data":"7507ec007258331b6205882a5fbf2571c41f72362e6d2e712c0a7850112275ba"} Oct 30 22:26:39.081976 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.081897 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7507ec007258331b6205882a5fbf2571c41f72362e6d2e712c0a7850112275ba" Oct 30 22:26:39.081976 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.081936 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb" Oct 30 22:26:39.088507 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.088488 4335 scope.go:122] "RemoveContainer" containerID="fb2bd9c0e515c45ccf7e4509858e4145c41f720232fd2bdc8c2869109a88c8be" Oct 30 22:26:39.088813 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:26:39.088791 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb2bd9c0e515c45ccf7e4509858e4145c41f720232fd2bdc8c2869109a88c8be\": container with ID starting with fb2bd9c0e515c45ccf7e4509858e4145c41f720232fd2bdc8c2869109a88c8be not found: ID does not exist" containerID="fb2bd9c0e515c45ccf7e4509858e4145c41f720232fd2bdc8c2869109a88c8be" Oct 30 22:26:39.088870 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.088819 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2bd9c0e515c45ccf7e4509858e4145c41f720232fd2bdc8c2869109a88c8be"} err="failed to get container status \"fb2bd9c0e515c45ccf7e4509858e4145c41f720232fd2bdc8c2869109a88c8be\": rpc error: code = NotFound desc = could not find container \"fb2bd9c0e515c45ccf7e4509858e4145c41f720232fd2bdc8c2869109a88c8be\": container with ID starting with fb2bd9c0e515c45ccf7e4509858e4145c41f720232fd2bdc8c2869109a88c8be not found: ID does not exist" Oct 30 22:26:39.088870 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.088834 4335 scope.go:122] "RemoveContainer" containerID="c71c6953b58a3b4625dd3f672eb249693875c14efcd5a497a719574a2246a442" Oct 30 22:26:39.135781 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.135754 4335 scope.go:122] "RemoveContainer" containerID="c71c6953b58a3b4625dd3f672eb249693875c14efcd5a497a719574a2246a442" Oct 30 22:26:39.136336 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:26:39.136310 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71c6953b58a3b4625dd3f672eb249693875c14efcd5a497a719574a2246a442\": container with ID starting with c71c6953b58a3b4625dd3f672eb249693875c14efcd5a497a719574a2246a442 not found: ID does not exist" containerID="c71c6953b58a3b4625dd3f672eb249693875c14efcd5a497a719574a2246a442" Oct 30 22:26:39.136467 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.136345 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71c6953b58a3b4625dd3f672eb249693875c14efcd5a497a719574a2246a442"} err="failed to get container status \"c71c6953b58a3b4625dd3f672eb249693875c14efcd5a497a719574a2246a442\": rpc error: code = NotFound desc = could not find container \"c71c6953b58a3b4625dd3f672eb249693875c14efcd5a497a719574a2246a442\": container with ID starting with c71c6953b58a3b4625dd3f672eb249693875c14efcd5a497a719574a2246a442 not found: ID does not exist" Oct 30 22:26:39.328709 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.324677 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx"] Oct 30 22:26:39.328709 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.328316 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-67657a-5855fd95dc-l4zdx"] Oct 30 22:26:39.539806 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.539769 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56"] Oct 30 22:26:39.543709 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.543681 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-67657a-7cf656b95c-l4j56"] Oct 30 22:26:39.594402 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.594337 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:26:39.595344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.595321 4335 patch_prober.go:29] interesting pod/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 30 22:26:39.595461 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:39.595370 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" podUID="1657e3aa-0cfe-49e5-b8ce-21888a334461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 30 22:26:40.593632 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:40.593590 4335 patch_prober.go:29] interesting pod/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 30 22:26:40.594138 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:40.593688 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" podUID="1657e3aa-0cfe-49e5-b8ce-21888a334461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 30 22:26:40.887181 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:40.887104 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:26:41.059915 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.059875 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b3173ec-26e6-4be5-9d48-911da303107b" path="/var/lib/kubelet/pods/0b3173ec-26e6-4be5-9d48-911da303107b/volumes" Oct 30 22:26:41.060396 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.060373 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2c9a56d-93df-4a25-bec4-6fff59fac92a" path="/var/lib/kubelet/pods/d2c9a56d-93df-4a25-bec4-6fff59fac92a/volumes" Oct 30 22:26:41.598639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.595031 4335 patch_prober.go:29] interesting pod/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 30 22:26:41.598639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.595088 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" podUID="1657e3aa-0cfe-49e5-b8ce-21888a334461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 30 22:26:41.736232 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.736202 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn"] Oct 30 22:26:41.736809 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.736794 4335 cpu_manager.go:414] "containerMap: removing container" podUID="d2c9a56d-93df-4a25-bec4-6fff59fac92a" containerName="payload-processing" Oct 30 22:26:41.736872 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.736811 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2c9a56d-93df-4a25-bec4-6fff59fac92a" containerName="payload-processing" Oct 30 22:26:41.736872 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.736823 4335 cpu_manager.go:414] "containerMap: removing container" podUID="97439aa6-0ced-4c9b-bd2e-73039917e317" containerName="revoke-keys" Oct 30 22:26:41.736872 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.736829 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97439aa6-0ced-4c9b-bd2e-73039917e317" containerName="revoke-keys" Oct 30 22:26:41.736872 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.736839 4335 cpu_manager.go:414] "containerMap: removing container" podUID="2f9df26f-a88b-431d-b6dc-d608694b1b55" containerName="revoke-keys" Oct 30 22:26:41.736872 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.736844 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f9df26f-a88b-431d-b6dc-d608694b1b55" containerName="revoke-keys" Oct 30 22:26:41.736872 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.736855 4335 cpu_manager.go:414] "containerMap: removing container" podUID="0b3173ec-26e6-4be5-9d48-911da303107b" containerName="payload-pre-processing" Oct 30 22:26:41.736872 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.736860 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b3173ec-26e6-4be5-9d48-911da303107b" containerName="payload-pre-processing" Oct 30 22:26:41.737099 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.736938 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f9df26f-a88b-431d-b6dc-d608694b1b55" containerName="revoke-keys" Oct 30 22:26:41.737099 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.736948 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b3173ec-26e6-4be5-9d48-911da303107b" containerName="payload-pre-processing" Oct 30 22:26:41.737099 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.736960 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2c9a56d-93df-4a25-bec4-6fff59fac92a" containerName="payload-processing" Oct 30 22:26:41.737099 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.736966 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="97439aa6-0ced-4c9b-bd2e-73039917e317" containerName="revoke-keys" Oct 30 22:26:41.740102 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.740085 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" Oct 30 22:26:41.792108 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.792083 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn"] Oct 30 22:26:41.816639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.799872 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-87b6b3\"" Oct 30 22:26:41.816639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.809344 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxbdg\" (UniqueName: \"kubernetes.io/projected/65723ec8-f97a-4172-86a5-9dfbffdea329-kube-api-access-fxbdg\") pod \"payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn\" (UID: \"65723ec8-f97a-4172-86a5-9dfbffdea329\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" Oct 30 22:26:41.816639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.809469 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65723ec8-f97a-4172-86a5-9dfbffdea329-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn\" (UID: \"65723ec8-f97a-4172-86a5-9dfbffdea329\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" Oct 30 22:26:41.842915 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.842881 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs"] Oct 30 22:26:41.846591 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.846572 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs" Oct 30 22:26:41.873431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.873408 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs"] Oct 30 22:26:41.927637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.926932 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-87b6b3-dockercfg-w696h\"" Oct 30 22:26:41.934049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.933810 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppnmr\" (UniqueName: \"kubernetes.io/projected/fc7b946c-d083-4b2f-829a-7066e9309ad5-kube-api-access-ppnmr\") pod \"payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs\" (UID: \"fc7b946c-d083-4b2f-829a-7066e9309ad5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs" Oct 30 22:26:41.934049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.933883 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxbdg\" (UniqueName: \"kubernetes.io/projected/65723ec8-f97a-4172-86a5-9dfbffdea329-kube-api-access-fxbdg\") pod \"payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn\" (UID: \"65723ec8-f97a-4172-86a5-9dfbffdea329\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" Oct 30 22:26:41.934049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.934014 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65723ec8-f97a-4172-86a5-9dfbffdea329-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn\" (UID: \"65723ec8-f97a-4172-86a5-9dfbffdea329\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" Oct 30 22:26:41.934330 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.934168 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7b946c-d083-4b2f-829a-7066e9309ad5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs\" (UID: \"fc7b946c-d083-4b2f-829a-7066e9309ad5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs" Oct 30 22:26:41.935114 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:41.935084 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65723ec8-f97a-4172-86a5-9dfbffdea329-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn\" (UID: \"65723ec8-f97a-4172-86a5-9dfbffdea329\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" Oct 30 22:26:42.008802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:42.008768 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxbdg\" (UniqueName: \"kubernetes.io/projected/65723ec8-f97a-4172-86a5-9dfbffdea329-kube-api-access-fxbdg\") pod \"payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn\" (UID: \"65723ec8-f97a-4172-86a5-9dfbffdea329\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" Oct 30 22:26:42.037067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:42.037027 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7b946c-d083-4b2f-829a-7066e9309ad5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs\" (UID: \"fc7b946c-d083-4b2f-829a-7066e9309ad5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs" Oct 30 22:26:42.038264 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:42.038149 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7b946c-d083-4b2f-829a-7066e9309ad5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs\" (UID: \"fc7b946c-d083-4b2f-829a-7066e9309ad5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs" Oct 30 22:26:42.038560 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:42.038537 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppnmr\" (UniqueName: \"kubernetes.io/projected/fc7b946c-d083-4b2f-829a-7066e9309ad5-kube-api-access-ppnmr\") pod \"payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs\" (UID: \"fc7b946c-d083-4b2f-829a-7066e9309ad5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs" Oct 30 22:26:42.061469 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:42.058226 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" Oct 30 22:26:42.122676 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:42.122292 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppnmr\" (UniqueName: \"kubernetes.io/projected/fc7b946c-d083-4b2f-829a-7066e9309ad5-kube-api-access-ppnmr\") pod \"payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs\" (UID: \"fc7b946c-d083-4b2f-829a-7066e9309ad5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs" Oct 30 22:26:42.162668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:42.162377 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs" Oct 30 22:26:42.540943 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:26:42.540889 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65723ec8_f97a_4172_86a5_9dfbffdea329.slice/crio-726cfc60484bc01011cce50c51c1023b240783b449d676a396a339640909310c WatchSource:0}: Error finding container 726cfc60484bc01011cce50c51c1023b240783b449d676a396a339640909310c: Status 404 returned error can't find the container with id 726cfc60484bc01011cce50c51c1023b240783b449d676a396a339640909310c Oct 30 22:26:42.543475 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:42.542931 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn"] Oct 30 22:26:42.594628 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:42.594583 4335 patch_prober.go:29] interesting pod/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 30 22:26:42.594749 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:42.594661 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" podUID="1657e3aa-0cfe-49e5-b8ce-21888a334461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 30 22:26:42.640603 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:42.640569 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs"] Oct 30 22:26:42.642976 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:26:42.642954 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7b946c_d083_4b2f_829a_7066e9309ad5.slice/crio-44ab1c10a18b9fadb256fc6406cbc1b9bd085524ad1f5a74cb7e680e7a12d8c8 WatchSource:0}: Error finding container 44ab1c10a18b9fadb256fc6406cbc1b9bd085524ad1f5a74cb7e680e7a12d8c8: Status 404 returned error can't find the container with id 44ab1c10a18b9fadb256fc6406cbc1b9bd085524ad1f5a74cb7e680e7a12d8c8 Oct 30 22:26:43.160803 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:43.160775 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" event={"ID":"65723ec8-f97a-4172-86a5-9dfbffdea329","Type":"ContainerStarted","Data":"c37fdf8be6e2a77262b6a9845e60d0536718becfc12b4079f072010f6654b721"} Oct 30 22:26:43.161038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:43.161023 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" event={"ID":"65723ec8-f97a-4172-86a5-9dfbffdea329","Type":"ContainerStarted","Data":"726cfc60484bc01011cce50c51c1023b240783b449d676a396a339640909310c"} Oct 30 22:26:43.161403 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:43.161388 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" Oct 30 22:26:43.162518 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:43.162502 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs" event={"ID":"fc7b946c-d083-4b2f-829a-7066e9309ad5","Type":"ContainerStarted","Data":"88e293f0bb280c74d04666eab9fdc4b900a62f91671592fc1913a1e6d9d70270"} Oct 30 22:26:43.162655 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:43.162643 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs" event={"ID":"fc7b946c-d083-4b2f-829a-7066e9309ad5","Type":"ContainerStarted","Data":"44ab1c10a18b9fadb256fc6406cbc1b9bd085524ad1f5a74cb7e680e7a12d8c8"} Oct 30 22:26:43.162741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:43.162734 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs" Oct 30 22:26:43.281091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:43.281044 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" podStartSLOduration=2.281031934 podStartE2EDuration="2.281031934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:43.276960438 +0000 UTC m=+2536.377392169" watchObservedRunningTime="2026-10-30 22:26:43.281031934 +0000 UTC m=+2536.381463667" Oct 30 22:26:43.396189 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:43.395168 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs" podStartSLOduration=2.395136655 podStartE2EDuration="2.395136655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:43.391894962 +0000 UTC m=+2536.492326695" watchObservedRunningTime="2026-10-30 22:26:43.395136655 +0000 UTC m=+2536.495568389" Oct 30 22:26:43.593974 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:43.593945 4335 patch_prober.go:29] interesting pod/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 30 22:26:43.594168 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:43.594007 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" podUID="1657e3aa-0cfe-49e5-b8ce-21888a334461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 30 22:26:43.830641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:43.829667 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf"] Oct 30 22:26:43.833569 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:43.833530 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7b06b-b8tzf"] Oct 30 22:26:44.649213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:44.648887 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:26:44.650725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:44.650689 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:26:45.060961 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:45.060923 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be962e0f-ae2d-4805-964e-fff48077329e" path="/var/lib/kubelet/pods/be962e0f-ae2d-4805-964e-fff48077329e/volumes" Oct 30 22:26:50.713658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:50.713606 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92"] Oct 30 22:26:50.714126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:50.713860 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" podUID="4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" containerName="istio-proxy" containerID="cri-o://319f24e86fa6d89dc354a0780de4cb3f4d04e99f39fcb66c3fed9f0f516662be" gracePeriod=30 Oct 30 22:26:51.068114 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.068085 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5"] Oct 30 22:26:51.072983 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.072962 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.151138 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.151089 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-32dd8d-openshift-default-dockercfg-m9gns\"" Oct 30 22:26:51.158099 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.158057 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5"] Oct 30 22:26:51.174602 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.174565 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-workload-socket\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.174906 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.174889 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-data\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.175195 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.175175 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-credential-socket\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.175381 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.175362 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-workload-certs\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.175542 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.175528 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-envoy\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.175676 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.175661 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-podinfo\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.175832 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.175797 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c280505e-e32e-4a17-9643-3be7e1250e0d-istiod-ca-cert\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.175954 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.175934 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-token\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.176478 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.176453 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhgmc\" (UniqueName: \"kubernetes.io/projected/c280505e-e32e-4a17-9643-3be7e1250e0d-kube-api-access-jhgmc\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.278435 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.278392 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-envoy\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.278703 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.278458 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-podinfo\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.278703 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.278489 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c280505e-e32e-4a17-9643-3be7e1250e0d-istiod-ca-cert\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.278703 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.278519 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-token\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.278703 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.278564 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhgmc\" (UniqueName: \"kubernetes.io/projected/c280505e-e32e-4a17-9643-3be7e1250e0d-kube-api-access-jhgmc\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.278703 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.278685 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-workload-socket\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.279006 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.278713 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-data\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.279006 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.278841 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-credential-socket\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.279006 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.278897 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-workload-certs\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.279419 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.279369 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-workload-certs\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.280063 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.280036 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-workload-socket\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.282441 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.282132 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c280505e-e32e-4a17-9643-3be7e1250e0d-istiod-ca-cert\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.282441 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.282420 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-credential-socket\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.283526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.282754 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-data\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.284361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.284297 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-podinfo\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.284932 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.284902 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-envoy\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.349319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.349256 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhgmc\" (UniqueName: \"kubernetes.io/projected/c280505e-e32e-4a17-9643-3be7e1250e0d-kube-api-access-jhgmc\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.353233 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.353212 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-token\") pod \"e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.385985 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.385959 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:51.820038 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:26:51.820004 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc280505e_e32e_4a17_9643_3be7e1250e0d.slice/crio-f9fff5d9bc1e214eb7559e1c7338d80f49aa2aa60a55d81111bbe9ff73477ef9 WatchSource:0}: Error finding container f9fff5d9bc1e214eb7559e1c7338d80f49aa2aa60a55d81111bbe9ff73477ef9: Status 404 returned error can't find the container with id f9fff5d9bc1e214eb7559e1c7338d80f49aa2aa60a55d81111bbe9ff73477ef9 Oct 30 22:26:51.820601 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.820580 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5"] Oct 30 22:26:51.822522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.822491 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:51.822597 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.822559 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:51.822597 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:51.822587 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:52.231118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:52.231039 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" event={"ID":"c280505e-e32e-4a17-9643-3be7e1250e0d","Type":"ContainerStarted","Data":"7a4207df4bfb0e9afb11dcf4ff44a831c5a9b1572bc0515f7380fbab7d61610a"} Oct 30 22:26:52.231118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:52.231096 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" event={"ID":"c280505e-e32e-4a17-9643-3be7e1250e0d","Type":"ContainerStarted","Data":"f9fff5d9bc1e214eb7559e1c7338d80f49aa2aa60a55d81111bbe9ff73477ef9"} Oct 30 22:26:52.354594 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:52.354544 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" podStartSLOduration=1.354530397 podStartE2EDuration="1.354530397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:52.35039253 +0000 UTC m=+2545.450824261" watchObservedRunningTime="2026-10-30 22:26:52.354530397 +0000 UTC m=+2545.454962128" Oct 30 22:26:52.386528 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:52.386502 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:52.391921 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:52.391897 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:53.240699 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:53.239149 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:53.240699 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:53.240195 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:26:54.173602 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:54.173569 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs" Oct 30 22:26:54.241431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:54.241397 4335 patch_prober.go:29] interesting pod/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:26:54.241926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:54.241455 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" podUID="4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:26:55.180775 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:55.180749 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" Oct 30 22:26:55.329007 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:26:55.328962 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:26:55.999301 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:55.999243 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 22:26:56.141437 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.141408 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-workload-certs\") pod \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " Oct 30 22:26:56.141660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.141457 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-token\" (UniqueName: \"kubernetes.io/projected/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-token\") pod \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " Oct 30 22:26:56.141660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.141481 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-credential-socket\") pod \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " Oct 30 22:26:56.141660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.141509 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-podinfo\") pod \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " Oct 30 22:26:56.141856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.141698 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-data\") pod \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " Oct 30 22:26:56.141856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.141723 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-envoy\") pod \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " Oct 30 22:26:56.141856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.141821 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-workload-certs" pod "4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" (UID: "4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:56.142029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.141940 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-data" pod "4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" (UID: "4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:56.142029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.141994 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-credential-socket" pod "4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" (UID: "4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:56.142179 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.142163 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-kube-api-access-xkgdz\" (UniqueName: \"kubernetes.io/projected/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-kube-api-access-xkgdz\") pod \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " Oct 30 22:26:56.142256 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.142230 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istiod-ca-cert\") pod \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " Oct 30 22:26:56.142325 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.142306 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-workload-socket\") pod \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\" (UID: \"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4\") " Oct 30 22:26:56.142795 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.142757 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istiod-ca-cert" pod "4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" (UID: "4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:56.142902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.142778 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:56.142965 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.142926 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:56.143023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.142966 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:56.143415 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.143383 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-workload-socket" pod "4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" (UID: "4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:56.144409 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.144384 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-token" pod "4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" (UID: "4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:56.145143 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.145113 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-kube-api-access-xkgdz" pod "4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" (UID: "4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4"). InnerVolumeSpecName "kube-api-access-xkgdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:56.145260 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.145193 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-envoy" pod "4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" (UID: "4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:56.145344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.145325 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-podinfo" pod "4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" (UID: "4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:56.244520 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.244496 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:56.244520 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.244519 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:56.244762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.244530 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:56.244762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.244538 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:56.244762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.244546 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkgdz\" (UniqueName: \"kubernetes.io/projected/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-kube-api-access-xkgdz\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:56.244762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.244555 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:56.264585 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.264527 4335 generic.go:356] "Generic (PLEG): container finished" podID="4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" containerID="319f24e86fa6d89dc354a0780de4cb3f4d04e99f39fcb66c3fed9f0f516662be" exitCode=0 Oct 30 22:26:56.264585 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.264581 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" Oct 30 22:26:56.264750 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.264668 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" event={"ID":"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4","Type":"ContainerDied","Data":"319f24e86fa6d89dc354a0780de4cb3f4d04e99f39fcb66c3fed9f0f516662be"} Oct 30 22:26:56.264750 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.264726 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92" event={"ID":"4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4","Type":"ContainerDied","Data":"741fe34af64372362e02ee4ed5de573a100ecb3b10f199c60b1f56676f8c4825"} Oct 30 22:26:56.264750 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.264748 4335 scope.go:122] "RemoveContainer" containerID="319f24e86fa6d89dc354a0780de4cb3f4d04e99f39fcb66c3fed9f0f516662be" Oct 30 22:26:56.276132 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.276117 4335 scope.go:122] "RemoveContainer" containerID="06b9a0c7bd9b0499ff12b761840ec9569aab185e82eabfa8d9e23ab9c1feb1f9" Oct 30 22:26:56.288699 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.288678 4335 scope.go:122] "RemoveContainer" containerID="319f24e86fa6d89dc354a0780de4cb3f4d04e99f39fcb66c3fed9f0f516662be" Oct 30 22:26:56.288957 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:26:56.288939 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319f24e86fa6d89dc354a0780de4cb3f4d04e99f39fcb66c3fed9f0f516662be\": container with ID starting with 319f24e86fa6d89dc354a0780de4cb3f4d04e99f39fcb66c3fed9f0f516662be not found: ID does not exist" containerID="319f24e86fa6d89dc354a0780de4cb3f4d04e99f39fcb66c3fed9f0f516662be" Oct 30 22:26:56.289014 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.288966 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319f24e86fa6d89dc354a0780de4cb3f4d04e99f39fcb66c3fed9f0f516662be"} err="failed to get container status \"319f24e86fa6d89dc354a0780de4cb3f4d04e99f39fcb66c3fed9f0f516662be\": rpc error: code = NotFound desc = could not find container \"319f24e86fa6d89dc354a0780de4cb3f4d04e99f39fcb66c3fed9f0f516662be\": container with ID starting with 319f24e86fa6d89dc354a0780de4cb3f4d04e99f39fcb66c3fed9f0f516662be not found: ID does not exist" Oct 30 22:26:56.289014 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.288983 4335 scope.go:122] "RemoveContainer" containerID="06b9a0c7bd9b0499ff12b761840ec9569aab185e82eabfa8d9e23ab9c1feb1f9" Oct 30 22:26:56.289213 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:26:56.289200 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b9a0c7bd9b0499ff12b761840ec9569aab185e82eabfa8d9e23ab9c1feb1f9\": container with ID starting with 06b9a0c7bd9b0499ff12b761840ec9569aab185e82eabfa8d9e23ab9c1feb1f9 not found: ID does not exist" containerID="06b9a0c7bd9b0499ff12b761840ec9569aab185e82eabfa8d9e23ab9c1feb1f9" Oct 30 22:26:56.289252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.289216 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b9a0c7bd9b0499ff12b761840ec9569aab185e82eabfa8d9e23ab9c1feb1f9"} err="failed to get container status \"06b9a0c7bd9b0499ff12b761840ec9569aab185e82eabfa8d9e23ab9c1feb1f9\": rpc error: code = NotFound desc = could not find container \"06b9a0c7bd9b0499ff12b761840ec9569aab185e82eabfa8d9e23ab9c1feb1f9\": container with ID starting with 06b9a0c7bd9b0499ff12b761840ec9569aab185e82eabfa8d9e23ab9c1feb1f9 not found: ID does not exist" Oct 30 22:26:56.432979 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.432952 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92"] Oct 30 22:26:56.436855 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:56.436835 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7b06b-openshift-default-d79c4d5dc-2kg92"] Oct 30 22:26:57.059096 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:57.059066 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" path="/var/lib/kubelet/pods/4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4/volumes" Oct 30 22:26:58.927896 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:58.927858 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q"] Oct 30 22:26:58.928498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:58.928431 4335 cpu_manager.go:414] "containerMap: removing container" podUID="4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" containerName="istio-proxy" Oct 30 22:26:58.928498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:58.928443 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" containerName="istio-proxy" Oct 30 22:26:58.928498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:58.928463 4335 cpu_manager.go:414] "containerMap: removing container" podUID="4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" containerName="istio-proxy" Oct 30 22:26:58.928498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:58.928468 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" containerName="istio-proxy" Oct 30 22:26:58.928775 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:58.928566 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" containerName="istio-proxy" Oct 30 22:26:58.928775 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:58.928576 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2eb14d-86a7-4cdb-919c-9a5d1296d5d4" containerName="istio-proxy" Oct 30 22:26:58.933541 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:58.933519 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" Oct 30 22:26:58.988910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:58.988875 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q"] Oct 30 22:26:59.053638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.053141 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-32dd8d-dockercfg-vfjt8\"" Oct 30 22:26:59.053638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.053205 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-32dd8d\"" Oct 30 22:26:59.075554 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.075270 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96dea67c-e674-4c4e-81d3-0c5bea462f7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q\" (UID: \"96dea67c-e674-4c4e-81d3-0c5bea462f7c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" Oct 30 22:26:59.076193 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.076168 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7gc6\" (UniqueName: \"kubernetes.io/projected/96dea67c-e674-4c4e-81d3-0c5bea462f7c-kube-api-access-g7gc6\") pod \"payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q\" (UID: \"96dea67c-e674-4c4e-81d3-0c5bea462f7c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" Oct 30 22:26:59.098556 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.097855 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk"] Oct 30 22:26:59.104375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.104340 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" Oct 30 22:26:59.152606 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.152566 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk"] Oct 30 22:26:59.178161 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.178098 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96dea67c-e674-4c4e-81d3-0c5bea462f7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q\" (UID: \"96dea67c-e674-4c4e-81d3-0c5bea462f7c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" Oct 30 22:26:59.178161 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.178152 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7gc6\" (UniqueName: \"kubernetes.io/projected/96dea67c-e674-4c4e-81d3-0c5bea462f7c-kube-api-access-g7gc6\") pod \"payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q\" (UID: \"96dea67c-e674-4c4e-81d3-0c5bea462f7c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" Oct 30 22:26:59.178959 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.178937 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96dea67c-e674-4c4e-81d3-0c5bea462f7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q\" (UID: \"96dea67c-e674-4c4e-81d3-0c5bea462f7c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" Oct 30 22:26:59.238999 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.238969 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7gc6\" (UniqueName: \"kubernetes.io/projected/96dea67c-e674-4c4e-81d3-0c5bea462f7c-kube-api-access-g7gc6\") pod \"payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q\" (UID: \"96dea67c-e674-4c4e-81d3-0c5bea462f7c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" Oct 30 22:26:59.248902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.248874 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" Oct 30 22:26:59.279525 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.279481 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp2fj\" (UniqueName: \"kubernetes.io/projected/0c240af7-e207-49f2-b539-8136924ff809-kube-api-access-pp2fj\") pod \"payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk\" (UID: \"0c240af7-e207-49f2-b539-8136924ff809\") " pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" Oct 30 22:26:59.279643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.279580 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c240af7-e207-49f2-b539-8136924ff809-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk\" (UID: \"0c240af7-e207-49f2-b539-8136924ff809\") " pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" Oct 30 22:26:59.380841 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.380811 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp2fj\" (UniqueName: \"kubernetes.io/projected/0c240af7-e207-49f2-b539-8136924ff809-kube-api-access-pp2fj\") pod \"payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk\" (UID: \"0c240af7-e207-49f2-b539-8136924ff809\") " pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" Oct 30 22:26:59.381003 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.380874 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c240af7-e207-49f2-b539-8136924ff809-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk\" (UID: \"0c240af7-e207-49f2-b539-8136924ff809\") " pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" Oct 30 22:26:59.381578 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.381558 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c240af7-e207-49f2-b539-8136924ff809-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk\" (UID: \"0c240af7-e207-49f2-b539-8136924ff809\") " pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" Oct 30 22:26:59.442057 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.441983 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp2fj\" (UniqueName: \"kubernetes.io/projected/0c240af7-e207-49f2-b539-8136924ff809-kube-api-access-pp2fj\") pod \"payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk\" (UID: \"0c240af7-e207-49f2-b539-8136924ff809\") " pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" Oct 30 22:26:59.632278 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.632248 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q"] Oct 30 22:26:59.633004 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:26:59.632976 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96dea67c_e674_4c4e_81d3_0c5bea462f7c.slice/crio-c0ba0b1fe24438903dbd2198e0132d67d292fccb94b87152c14a859113dbc6c2 WatchSource:0}: Error finding container c0ba0b1fe24438903dbd2198e0132d67d292fccb94b87152c14a859113dbc6c2: Status 404 returned error can't find the container with id c0ba0b1fe24438903dbd2198e0132d67d292fccb94b87152c14a859113dbc6c2 Oct 30 22:26:59.724557 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:26:59.724486 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" Oct 30 22:27:00.107217 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:00.107192 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk"] Oct 30 22:27:00.108871 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:27:00.108840 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c240af7_e207_49f2_b539_8136924ff809.slice/crio-e150ac3a9b527a9bbe74d0abb7452c33fdd2a2900159954cdc0012e9d8c7d818 WatchSource:0}: Error finding container e150ac3a9b527a9bbe74d0abb7452c33fdd2a2900159954cdc0012e9d8c7d818: Status 404 returned error can't find the container with id e150ac3a9b527a9bbe74d0abb7452c33fdd2a2900159954cdc0012e9d8c7d818 Oct 30 22:27:00.298394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:00.298356 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" event={"ID":"96dea67c-e674-4c4e-81d3-0c5bea462f7c","Type":"ContainerStarted","Data":"e75178bdcc15529ebecefd7a726478ee5f7a856d6aa12d8797950fe371fcf045"} Oct 30 22:27:00.298578 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:00.298421 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" event={"ID":"96dea67c-e674-4c4e-81d3-0c5bea462f7c","Type":"ContainerStarted","Data":"c0ba0b1fe24438903dbd2198e0132d67d292fccb94b87152c14a859113dbc6c2"} Oct 30 22:27:00.298578 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:00.298451 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" Oct 30 22:27:00.300472 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:00.300448 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" event={"ID":"0c240af7-e207-49f2-b539-8136924ff809","Type":"ContainerStarted","Data":"98bcf501b1440d508565bcf37113b1a0d1d6c0d1e00aee8d65fd045623b93a47"} Oct 30 22:27:00.300592 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:00.300493 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" event={"ID":"0c240af7-e207-49f2-b539-8136924ff809","Type":"ContainerStarted","Data":"e150ac3a9b527a9bbe74d0abb7452c33fdd2a2900159954cdc0012e9d8c7d818"} Oct 30 22:27:00.300592 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:00.300571 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" Oct 30 22:27:00.416034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:00.415950 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" podStartSLOduration=2.415936223 podStartE2EDuration="2.415936223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:00.412558662 +0000 UTC m=+2553.512990395" watchObservedRunningTime="2026-10-30 22:27:00.415936223 +0000 UTC m=+2553.516367954" Oct 30 22:27:00.532350 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:00.532293 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" podStartSLOduration=1.5322745530000002 podStartE2EDuration="1.532274553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:00.531530284 +0000 UTC m=+2553.631962016" watchObservedRunningTime="2026-10-30 22:27:00.532274553 +0000 UTC m=+2553.632706289" Oct 30 22:27:03.919602 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:03.919571 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:27:11.311320 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:11.311244 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" Oct 30 22:27:11.311320 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:11.311308 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" Oct 30 22:27:20.572877 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:20.572846 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg"] Oct 30 22:27:20.622384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:20.622350 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cef074-p84bg"] Oct 30 22:27:21.057923 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:21.057895 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f6c5ae4-4d9f-4efe-a696-a0780affb25d" path="/var/lib/kubelet/pods/4f6c5ae4-4d9f-4efe-a696-a0780affb25d/volumes" Oct 30 22:27:21.857851 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:21.857813 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq"] Oct 30 22:27:21.906496 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:21.906466 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-67657a-j7sbq"] Oct 30 22:27:23.057466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:23.057432 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81201e64-6a4c-40d5-9d17-3b4cf575abd3" path="/var/lib/kubelet/pods/81201e64-6a4c-40d5-9d17-3b4cf575abd3/volumes" Oct 30 22:27:25.320690 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:27:25.320647 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:27:40.454957 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:40.454919 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf"] Oct 30 22:27:40.461012 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:40.460992 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf" Oct 30 22:27:40.522415 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:40.522383 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf"] Oct 30 22:27:40.565428 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:40.565387 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:27:40.565634 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:40.565451 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:27:40.565634 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:40.565452 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:27:40.565634 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:40.565533 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:27:40.656016 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:40.655986 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63d6602-8612-47cf-9174-a873eaf712b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf\" (UID: \"e63d6602-8612-47cf-9174-a873eaf712b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf" Oct 30 22:27:40.656213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:40.656137 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tpgc\" (UniqueName: \"kubernetes.io/projected/e63d6602-8612-47cf-9174-a873eaf712b0-kube-api-access-6tpgc\") pod \"maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf\" (UID: \"e63d6602-8612-47cf-9174-a873eaf712b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf" Oct 30 22:27:40.656213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:40.656182 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63d6602-8612-47cf-9174-a873eaf712b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf\" (UID: \"e63d6602-8612-47cf-9174-a873eaf712b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf" Oct 30 22:27:40.757857 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:40.757784 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tpgc\" (UniqueName: \"kubernetes.io/projected/e63d6602-8612-47cf-9174-a873eaf712b0-kube-api-access-6tpgc\") pod \"maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf\" (UID: \"e63d6602-8612-47cf-9174-a873eaf712b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf" Oct 30 22:27:40.757857 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:40.757831 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63d6602-8612-47cf-9174-a873eaf712b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf\" (UID: \"e63d6602-8612-47cf-9174-a873eaf712b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf" Oct 30 22:27:40.758044 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:40.757910 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63d6602-8612-47cf-9174-a873eaf712b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf\" (UID: \"e63d6602-8612-47cf-9174-a873eaf712b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf" Oct 30 22:27:40.758560 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:40.758538 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63d6602-8612-47cf-9174-a873eaf712b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf\" (UID: \"e63d6602-8612-47cf-9174-a873eaf712b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf" Oct 30 22:27:40.760536 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:40.760510 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63d6602-8612-47cf-9174-a873eaf712b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf\" (UID: \"e63d6602-8612-47cf-9174-a873eaf712b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf" Oct 30 22:27:40.828427 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:40.828395 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tpgc\" (UniqueName: \"kubernetes.io/projected/e63d6602-8612-47cf-9174-a873eaf712b0-kube-api-access-6tpgc\") pod \"maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf\" (UID: \"e63d6602-8612-47cf-9174-a873eaf712b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf" Oct 30 22:27:41.074572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:41.074542 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf" Oct 30 22:27:41.493022 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:41.492993 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf"] Oct 30 22:27:41.494157 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:27:41.494130 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63d6602_8612_47cf_9174_a873eaf712b0.slice/crio-5298aa369e18aeb7d5bfbd09c0dd7dc2712ded22f117ff7c342c2f3235283327 WatchSource:0}: Error finding container 5298aa369e18aeb7d5bfbd09c0dd7dc2712ded22f117ff7c342c2f3235283327: Status 404 returned error can't find the container with id 5298aa369e18aeb7d5bfbd09c0dd7dc2712ded22f117ff7c342c2f3235283327 Oct 30 22:27:41.661795 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:41.661763 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf" event={"ID":"e63d6602-8612-47cf-9174-a873eaf712b0","Type":"ContainerStarted","Data":"5298aa369e18aeb7d5bfbd09c0dd7dc2712ded22f117ff7c342c2f3235283327"} Oct 30 22:27:42.672364 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:42.672325 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf" event={"ID":"e63d6602-8612-47cf-9174-a873eaf712b0","Type":"ContainerStarted","Data":"4384d9560dd1b427eb6fff46c07f7d9eb4b4e922a39e84cf3585cb396d2b9d18"} Oct 30 22:27:42.791878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:42.791825 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf" podStartSLOduration=2.791811348 podStartE2EDuration="2.791811348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:42.787986744 +0000 UTC m=+2595.888418477" watchObservedRunningTime="2026-10-30 22:27:42.791811348 +0000 UTC m=+2595.892243079" Oct 30 22:27:43.682912 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:43.682871 4335 generic.go:356] "Generic (PLEG): container finished" podID="e63d6602-8612-47cf-9174-a873eaf712b0" containerID="4384d9560dd1b427eb6fff46c07f7d9eb4b4e922a39e84cf3585cb396d2b9d18" exitCode=0 Oct 30 22:27:43.683307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:43.682944 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf" event={"ID":"e63d6602-8612-47cf-9174-a873eaf712b0","Type":"ContainerDied","Data":"4384d9560dd1b427eb6fff46c07f7d9eb4b4e922a39e84cf3585cb396d2b9d18"} Oct 30 22:27:44.878979 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:44.878954 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf" Oct 30 22:27:45.039894 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:45.039804 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e63d6602-8612-47cf-9174-a873eaf712b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63d6602-8612-47cf-9174-a873eaf712b0-maas-api-openshift-ca\") pod \"e63d6602-8612-47cf-9174-a873eaf712b0\" (UID: \"e63d6602-8612-47cf-9174-a873eaf712b0\") " Oct 30 22:27:45.039894 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:45.039886 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63d6602-8612-47cf-9174-a873eaf712b0-kube-api-access-6tpgc\" (UniqueName: \"kubernetes.io/projected/e63d6602-8612-47cf-9174-a873eaf712b0-kube-api-access-6tpgc\") pod \"e63d6602-8612-47cf-9174-a873eaf712b0\" (UID: \"e63d6602-8612-47cf-9174-a873eaf712b0\") " Oct 30 22:27:45.040160 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:45.040128 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e63d6602-8612-47cf-9174-a873eaf712b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63d6602-8612-47cf-9174-a873eaf712b0-maas-api-xks-ca\") pod \"e63d6602-8612-47cf-9174-a873eaf712b0\" (UID: \"e63d6602-8612-47cf-9174-a873eaf712b0\") " Oct 30 22:27:45.051636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:45.050237 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63d6602-8612-47cf-9174-a873eaf712b0-maas-api-openshift-ca" pod "e63d6602-8612-47cf-9174-a873eaf712b0" (UID: "e63d6602-8612-47cf-9174-a873eaf712b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:45.070329 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:45.070271 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e63d6602-8612-47cf-9174-a873eaf712b0-maas-api-xks-ca" pod "e63d6602-8612-47cf-9174-a873eaf712b0" (UID: "e63d6602-8612-47cf-9174-a873eaf712b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:45.083638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:45.081203 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63d6602-8612-47cf-9174-a873eaf712b0-kube-api-access-6tpgc" pod "e63d6602-8612-47cf-9174-a873eaf712b0" (UID: "e63d6602-8612-47cf-9174-a873eaf712b0"). InnerVolumeSpecName "kube-api-access-6tpgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:45.142309 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:45.142278 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63d6602-8612-47cf-9174-a873eaf712b0-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:45.142309 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:45.142307 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tpgc\" (UniqueName: \"kubernetes.io/projected/e63d6602-8612-47cf-9174-a873eaf712b0-kube-api-access-6tpgc\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:45.142526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:45.142317 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63d6602-8612-47cf-9174-a873eaf712b0-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:45.709179 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:45.709147 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf" Oct 30 22:27:45.709352 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:45.709180 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf" event={"ID":"e63d6602-8612-47cf-9174-a873eaf712b0","Type":"ContainerDied","Data":"5298aa369e18aeb7d5bfbd09c0dd7dc2712ded22f117ff7c342c2f3235283327"} Oct 30 22:27:45.709352 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:45.709228 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5298aa369e18aeb7d5bfbd09c0dd7dc2712ded22f117ff7c342c2f3235283327" Oct 30 22:27:46.720075 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:46.719990 4335 generic.go:356] "Generic (PLEG): container finished" podID="f2ad9d54-24dc-40ea-9397-5bfc51b9d786" containerID="8cba0bc44621f7c2cec491e32470b93bc6487a4773f2cf7f0fd490b4ead19284" exitCode=0 Oct 30 22:27:46.720523 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:46.720065 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" event={"ID":"f2ad9d54-24dc-40ea-9397-5bfc51b9d786","Type":"ContainerDied","Data":"8cba0bc44621f7c2cec491e32470b93bc6487a4773f2cf7f0fd490b4ead19284"} Oct 30 22:27:46.720523 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:46.720450 4335 scope.go:122] "RemoveContainer" containerID="8cba0bc44621f7c2cec491e32470b93bc6487a4773f2cf7f0fd490b4ead19284" Oct 30 22:27:46.722066 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:46.722031 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:27:46.722197 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:46.722104 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:27:46.722197 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:46.722145 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:27:47.731387 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:47.731353 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" event={"ID":"f2ad9d54-24dc-40ea-9397-5bfc51b9d786","Type":"ContainerStarted","Data":"1a7dbe29556bde2ef478f56041ac7a7733557d2b05ed7a749ae5acf9707af41e"} Oct 30 22:27:48.425606 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:48.425576 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:27:48.427151 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:48.427126 4335 patch_prober.go:29] interesting pod/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:27:48.427349 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:48.427173 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" podUID="f2ad9d54-24dc-40ea-9397-5bfc51b9d786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 30 22:27:49.427424 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:49.426966 4335 patch_prober.go:29] interesting pod/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:27:49.427424 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:49.427043 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" podUID="f2ad9d54-24dc-40ea-9397-5bfc51b9d786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 30 22:27:50.425353 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:50.425316 4335 patch_prober.go:29] interesting pod/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:27:50.425565 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:50.425378 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" podUID="f2ad9d54-24dc-40ea-9397-5bfc51b9d786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 30 22:27:50.844670 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:50.844634 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj"] Oct 30 22:27:50.963292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:50.963262 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q"] Oct 30 22:27:51.021745 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:51.021059 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:27:51.425998 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:51.425961 4335 patch_prober.go:29] interesting pod/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:27:51.426329 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:51.426300 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" podUID="f2ad9d54-24dc-40ea-9397-5bfc51b9d786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 30 22:27:52.428046 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:52.428003 4335 patch_prober.go:29] interesting pod/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 30 22:27:52.428754 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:52.428720 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" podUID="f2ad9d54-24dc-40ea-9397-5bfc51b9d786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 30 22:27:52.989581 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:52.989537 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp"] Oct 30 22:27:52.990178 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:52.990162 4335 cpu_manager.go:414] "containerMap: removing container" podUID="e63d6602-8612-47cf-9174-a873eaf712b0" containerName="revoke-keys" Oct 30 22:27:52.990228 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:52.990179 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63d6602-8612-47cf-9174-a873eaf712b0" containerName="revoke-keys" Oct 30 22:27:52.990282 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:52.990273 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63d6602-8612-47cf-9174-a873eaf712b0" containerName="revoke-keys" Oct 30 22:27:52.993988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:52.993971 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp" Oct 30 22:27:53.064366 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.064333 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93708d6a-0a8e-4730-86ef-eb02f3eca95d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp\" (UID: \"93708d6a-0a8e-4730-86ef-eb02f3eca95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp" Oct 30 22:27:53.064535 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.064477 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93708d6a-0a8e-4730-86ef-eb02f3eca95d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp\" (UID: \"93708d6a-0a8e-4730-86ef-eb02f3eca95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp" Oct 30 22:27:53.064583 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.064555 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnp5k\" (UniqueName: \"kubernetes.io/projected/93708d6a-0a8e-4730-86ef-eb02f3eca95d-kube-api-access-wnp5k\") pod \"maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp\" (UID: \"93708d6a-0a8e-4730-86ef-eb02f3eca95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp" Oct 30 22:27:53.096680 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.096642 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:27:53.096866 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.096849 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:27:53.096944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.096932 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:27:53.097472 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.096853 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:27:53.101175 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.101144 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp"] Oct 30 22:27:53.165401 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.165374 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnp5k\" (UniqueName: \"kubernetes.io/projected/93708d6a-0a8e-4730-86ef-eb02f3eca95d-kube-api-access-wnp5k\") pod \"maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp\" (UID: \"93708d6a-0a8e-4730-86ef-eb02f3eca95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp" Oct 30 22:27:53.165568 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.165487 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93708d6a-0a8e-4730-86ef-eb02f3eca95d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp\" (UID: \"93708d6a-0a8e-4730-86ef-eb02f3eca95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp" Oct 30 22:27:53.165568 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.165543 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93708d6a-0a8e-4730-86ef-eb02f3eca95d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp\" (UID: \"93708d6a-0a8e-4730-86ef-eb02f3eca95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp" Oct 30 22:27:53.166161 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.166130 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93708d6a-0a8e-4730-86ef-eb02f3eca95d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp\" (UID: \"93708d6a-0a8e-4730-86ef-eb02f3eca95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp" Oct 30 22:27:53.168046 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.168026 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93708d6a-0a8e-4730-86ef-eb02f3eca95d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp\" (UID: \"93708d6a-0a8e-4730-86ef-eb02f3eca95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp" Oct 30 22:27:53.226372 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.226346 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnp5k\" (UniqueName: \"kubernetes.io/projected/93708d6a-0a8e-4730-86ef-eb02f3eca95d-kube-api-access-wnp5k\") pod \"maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp\" (UID: \"93708d6a-0a8e-4730-86ef-eb02f3eca95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp" Oct 30 22:27:53.309869 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.309847 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp" Oct 30 22:27:53.430426 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.430400 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:27:53.431704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.431685 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:27:53.719500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.719466 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp"] Oct 30 22:27:53.783920 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:53.783893 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp" event={"ID":"93708d6a-0a8e-4730-86ef-eb02f3eca95d","Type":"ContainerStarted","Data":"1e30f5392d1069cf8121f6c0d5473b8ad14e101529c39ccdedc309a3e766f256"} Oct 30 22:27:54.793039 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:54.793008 4335 generic.go:356] "Generic (PLEG): container finished" podID="93708d6a-0a8e-4730-86ef-eb02f3eca95d" containerID="d2ddf45c8c30153b1e6800515015ebe71c539b07ecb5d4b31fe3514622dcc1f3" exitCode=0 Oct 30 22:27:54.793391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:54.793094 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp" event={"ID":"93708d6a-0a8e-4730-86ef-eb02f3eca95d","Type":"ContainerDied","Data":"d2ddf45c8c30153b1e6800515015ebe71c539b07ecb5d4b31fe3514622dcc1f3"} Oct 30 22:27:55.322254 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:27:55.322218 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:27:55.845601 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:55.845555 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" podUID="ca1f40ea-c3a3-4522-b526-fb3aad6183fa" containerName="payload-processing" containerID="cri-o://fde7e5a9aed94d157063a00d2e6bbe997f6916935068b72a84f500aa331eef6b" gracePeriod=25 Oct 30 22:27:55.963514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:55.963482 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" podUID="9bbb21e7-621e-4490-bc0b-f31ee5b34404" containerName="payload-pre-processing" containerID="cri-o://54f53254f640eb49e4801bdeb5dc98c0b1647485b077e8648873a3938eee7721" gracePeriod=25 Oct 30 22:27:55.963738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:55.963721 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp" Oct 30 22:27:55.994974 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:55.994942 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93708d6a-0a8e-4730-86ef-eb02f3eca95d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93708d6a-0a8e-4730-86ef-eb02f3eca95d-maas-api-xks-ca\") pod \"93708d6a-0a8e-4730-86ef-eb02f3eca95d\" (UID: \"93708d6a-0a8e-4730-86ef-eb02f3eca95d\") " Oct 30 22:27:55.995180 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:55.995053 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93708d6a-0a8e-4730-86ef-eb02f3eca95d-kube-api-access-wnp5k\" (UniqueName: \"kubernetes.io/projected/93708d6a-0a8e-4730-86ef-eb02f3eca95d-kube-api-access-wnp5k\") pod \"93708d6a-0a8e-4730-86ef-eb02f3eca95d\" (UID: \"93708d6a-0a8e-4730-86ef-eb02f3eca95d\") " Oct 30 22:27:55.995180 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:55.995172 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93708d6a-0a8e-4730-86ef-eb02f3eca95d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93708d6a-0a8e-4730-86ef-eb02f3eca95d-maas-api-openshift-ca\") pod \"93708d6a-0a8e-4730-86ef-eb02f3eca95d\" (UID: \"93708d6a-0a8e-4730-86ef-eb02f3eca95d\") " Oct 30 22:27:55.996024 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:55.995996 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93708d6a-0a8e-4730-86ef-eb02f3eca95d-maas-api-openshift-ca" pod "93708d6a-0a8e-4730-86ef-eb02f3eca95d" (UID: "93708d6a-0a8e-4730-86ef-eb02f3eca95d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:55.997999 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:55.997974 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93708d6a-0a8e-4730-86ef-eb02f3eca95d-kube-api-access-wnp5k" pod "93708d6a-0a8e-4730-86ef-eb02f3eca95d" (UID: "93708d6a-0a8e-4730-86ef-eb02f3eca95d"). InnerVolumeSpecName "kube-api-access-wnp5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:55.998119 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:55.998003 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93708d6a-0a8e-4730-86ef-eb02f3eca95d-maas-api-xks-ca" pod "93708d6a-0a8e-4730-86ef-eb02f3eca95d" (UID: "93708d6a-0a8e-4730-86ef-eb02f3eca95d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:56.097057 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:56.096995 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnp5k\" (UniqueName: \"kubernetes.io/projected/93708d6a-0a8e-4730-86ef-eb02f3eca95d-kube-api-access-wnp5k\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:56.097057 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:56.097022 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93708d6a-0a8e-4730-86ef-eb02f3eca95d-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:56.097057 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:56.097032 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93708d6a-0a8e-4730-86ef-eb02f3eca95d-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:56.811837 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:56.811801 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp" Oct 30 22:27:56.811837 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:56.811820 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp" event={"ID":"93708d6a-0a8e-4730-86ef-eb02f3eca95d","Type":"ContainerDied","Data":"1e30f5392d1069cf8121f6c0d5473b8ad14e101529c39ccdedc309a3e766f256"} Oct 30 22:27:56.812060 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:56.811875 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e30f5392d1069cf8121f6c0d5473b8ad14e101529c39ccdedc309a3e766f256" Oct 30 22:27:57.406102 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:57.406068 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh"] Oct 30 22:27:58.369094 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:27:58.369054 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:28:00.941431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:00.941395 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" podUID="ca1f40ea-c3a3-4522-b526-fb3aad6183fa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:01.949423 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:01.949380 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" podUID="9bbb21e7-621e-4490-bc0b-f31ee5b34404" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:03.450532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:03.450493 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn"] Oct 30 22:28:03.557841 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:03.557806 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j"] Oct 30 22:28:07.415887 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.415850 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f"] Oct 30 22:28:07.416801 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.416778 4335 cpu_manager.go:414] "containerMap: removing container" podUID="93708d6a-0a8e-4730-86ef-eb02f3eca95d" containerName="revoke-keys" Oct 30 22:28:07.416863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.416803 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93708d6a-0a8e-4730-86ef-eb02f3eca95d" containerName="revoke-keys" Oct 30 22:28:07.416980 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.416966 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="93708d6a-0a8e-4730-86ef-eb02f3eca95d" containerName="revoke-keys" Oct 30 22:28:07.421466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.421446 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f" Oct 30 22:28:07.475790 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.475751 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f"] Oct 30 22:28:07.518995 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.518962 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:28:07.519187 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.518989 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:28:07.522552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.522527 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:28:07.522552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.522547 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:28:07.558860 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.558829 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llj6x\" (UniqueName: \"kubernetes.io/projected/bd40b3cc-a54a-4d37-957e-40715570dbbf-kube-api-access-llj6x\") pod \"maas-api-revoke-sub-2a87cf99-c6c6f\" (UID: \"bd40b3cc-a54a-4d37-957e-40715570dbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f" Oct 30 22:28:07.558985 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.558943 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd40b3cc-a54a-4d37-957e-40715570dbbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a87cf99-c6c6f\" (UID: \"bd40b3cc-a54a-4d37-957e-40715570dbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f" Oct 30 22:28:07.558985 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.558976 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd40b3cc-a54a-4d37-957e-40715570dbbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a87cf99-c6c6f\" (UID: \"bd40b3cc-a54a-4d37-957e-40715570dbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f" Oct 30 22:28:07.660318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.660285 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llj6x\" (UniqueName: \"kubernetes.io/projected/bd40b3cc-a54a-4d37-957e-40715570dbbf-kube-api-access-llj6x\") pod \"maas-api-revoke-sub-2a87cf99-c6c6f\" (UID: \"bd40b3cc-a54a-4d37-957e-40715570dbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f" Oct 30 22:28:07.660488 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.660422 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd40b3cc-a54a-4d37-957e-40715570dbbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a87cf99-c6c6f\" (UID: \"bd40b3cc-a54a-4d37-957e-40715570dbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f" Oct 30 22:28:07.660488 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.660469 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd40b3cc-a54a-4d37-957e-40715570dbbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a87cf99-c6c6f\" (UID: \"bd40b3cc-a54a-4d37-957e-40715570dbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f" Oct 30 22:28:07.661077 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.661052 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd40b3cc-a54a-4d37-957e-40715570dbbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a87cf99-c6c6f\" (UID: \"bd40b3cc-a54a-4d37-957e-40715570dbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f" Oct 30 22:28:07.662983 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.662957 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd40b3cc-a54a-4d37-957e-40715570dbbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a87cf99-c6c6f\" (UID: \"bd40b3cc-a54a-4d37-957e-40715570dbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f" Oct 30 22:28:07.721215 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.721162 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llj6x\" (UniqueName: \"kubernetes.io/projected/bd40b3cc-a54a-4d37-957e-40715570dbbf-kube-api-access-llj6x\") pod \"maas-api-revoke-sub-2a87cf99-c6c6f\" (UID: \"bd40b3cc-a54a-4d37-957e-40715570dbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f" Oct 30 22:28:07.743099 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:07.743075 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f" Oct 30 22:28:08.144378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:08.144342 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f"] Oct 30 22:28:08.144709 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:28:08.144680 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd40b3cc_a54a_4d37_957e_40715570dbbf.slice/crio-0e3965ccb4e186a99eec7e6aa30c42684848227c28f281c14d89d5c49e3cade0 WatchSource:0}: Error finding container 0e3965ccb4e186a99eec7e6aa30c42684848227c28f281c14d89d5c49e3cade0: Status 404 returned error can't find the container with id 0e3965ccb4e186a99eec7e6aa30c42684848227c28f281c14d89d5c49e3cade0 Oct 30 22:28:08.450935 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:08.450904 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" podUID="8a7601c3-d83d-4d7f-b5d3-7528cba61d67" containerName="payload-processing" containerID="cri-o://db3836b80dd561fdf8585ab351e6e3a3d9211908021d520c7af2c91b8e8b740a" gracePeriod=25 Oct 30 22:28:08.502104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:08.502073 4335 patch_prober.go:29] interesting pod/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": read tcp 10.133.0.2:33018->10.133.0.247:15021: read: connection reset by peer" start-of-body= Oct 30 22:28:08.502247 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:08.502125 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" podUID="c280505e-e32e-4a17-9643-3be7e1250e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": read tcp 10.133.0.2:33018->10.133.0.247:15021: read: connection reset by peer" Oct 30 22:28:08.558577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:08.558543 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" podUID="f85380bc-9459-4ef3-94c1-663683361092" containerName="payload-pre-processing" containerID="cri-o://6c22a1475ab2a012cdf840909d2814a8ab103f4c6afb739adfc722cdb120ecba" gracePeriod=25 Oct 30 22:28:08.926696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:08.926596 4335 generic.go:356] "Generic (PLEG): container finished" podID="c280505e-e32e-4a17-9643-3be7e1250e0d" containerID="7a4207df4bfb0e9afb11dcf4ff44a831c5a9b1572bc0515f7380fbab7d61610a" exitCode=0 Oct 30 22:28:08.926696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:08.926647 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" event={"ID":"c280505e-e32e-4a17-9643-3be7e1250e0d","Type":"ContainerDied","Data":"7a4207df4bfb0e9afb11dcf4ff44a831c5a9b1572bc0515f7380fbab7d61610a"} Oct 30 22:28:08.927045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:08.927030 4335 scope.go:122] "RemoveContainer" containerID="7a4207df4bfb0e9afb11dcf4ff44a831c5a9b1572bc0515f7380fbab7d61610a" Oct 30 22:28:08.928609 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:08.928578 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:28:08.928766 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:08.928705 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:28:08.928766 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:08.928754 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:28:08.929372 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:08.929346 4335 generic.go:356] "Generic (PLEG): container finished" podID="bd40b3cc-a54a-4d37-957e-40715570dbbf" containerID="2eedc53b98ef41988f0f02c643f1c4a05c214be813b7762316dc12d3f220aba2" exitCode=0 Oct 30 22:28:08.929462 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:08.929373 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f" event={"ID":"bd40b3cc-a54a-4d37-957e-40715570dbbf","Type":"ContainerDied","Data":"2eedc53b98ef41988f0f02c643f1c4a05c214be813b7762316dc12d3f220aba2"} Oct 30 22:28:08.929462 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:08.929416 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f" event={"ID":"bd40b3cc-a54a-4d37-957e-40715570dbbf","Type":"ContainerStarted","Data":"0e3965ccb4e186a99eec7e6aa30c42684848227c28f281c14d89d5c49e3cade0"} Oct 30 22:28:09.941950 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:09.941918 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" event={"ID":"c280505e-e32e-4a17-9643-3be7e1250e0d","Type":"ContainerStarted","Data":"427e922cc7671ba9a8af3d009d4cb7fc8bc4c650a187554c0c9258a7373d07ba"} Oct 30 22:28:10.167296 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:10.167274 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f" Oct 30 22:28:10.292155 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:10.292075 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd40b3cc-a54a-4d37-957e-40715570dbbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd40b3cc-a54a-4d37-957e-40715570dbbf-maas-api-openshift-ca\") pod \"bd40b3cc-a54a-4d37-957e-40715570dbbf\" (UID: \"bd40b3cc-a54a-4d37-957e-40715570dbbf\") " Oct 30 22:28:10.292338 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:10.292242 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd40b3cc-a54a-4d37-957e-40715570dbbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd40b3cc-a54a-4d37-957e-40715570dbbf-maas-api-xks-ca\") pod \"bd40b3cc-a54a-4d37-957e-40715570dbbf\" (UID: \"bd40b3cc-a54a-4d37-957e-40715570dbbf\") " Oct 30 22:28:10.292430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:10.292409 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd40b3cc-a54a-4d37-957e-40715570dbbf-kube-api-access-llj6x\" (UniqueName: \"kubernetes.io/projected/bd40b3cc-a54a-4d37-957e-40715570dbbf-kube-api-access-llj6x\") pod \"bd40b3cc-a54a-4d37-957e-40715570dbbf\" (UID: \"bd40b3cc-a54a-4d37-957e-40715570dbbf\") " Oct 30 22:28:10.292493 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:10.292421 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd40b3cc-a54a-4d37-957e-40715570dbbf-maas-api-openshift-ca" pod "bd40b3cc-a54a-4d37-957e-40715570dbbf" (UID: "bd40b3cc-a54a-4d37-957e-40715570dbbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:10.292946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:10.292916 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd40b3cc-a54a-4d37-957e-40715570dbbf-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:10.294802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:10.294774 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd40b3cc-a54a-4d37-957e-40715570dbbf-maas-api-xks-ca" pod "bd40b3cc-a54a-4d37-957e-40715570dbbf" (UID: "bd40b3cc-a54a-4d37-957e-40715570dbbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:10.295236 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:10.295204 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd40b3cc-a54a-4d37-957e-40715570dbbf-kube-api-access-llj6x" pod "bd40b3cc-a54a-4d37-957e-40715570dbbf" (UID: "bd40b3cc-a54a-4d37-957e-40715570dbbf"). InnerVolumeSpecName "kube-api-access-llj6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:10.386586 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:10.386556 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:28:10.386849 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:10.386824 4335 patch_prober.go:29] interesting pod/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 22:28:10.386906 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:10.386883 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" podUID="c280505e-e32e-4a17-9643-3be7e1250e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 22:28:10.394515 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:10.394495 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llj6x\" (UniqueName: \"kubernetes.io/projected/bd40b3cc-a54a-4d37-957e-40715570dbbf-kube-api-access-llj6x\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:10.394593 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:10.394518 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd40b3cc-a54a-4d37-957e-40715570dbbf-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:10.941684 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:10.941651 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" podUID="ca1f40ea-c3a3-4522-b526-fb3aad6183fa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:10.951846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:10.951820 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f" Oct 30 22:28:10.952252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:10.951821 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f" event={"ID":"bd40b3cc-a54a-4d37-957e-40715570dbbf","Type":"ContainerDied","Data":"0e3965ccb4e186a99eec7e6aa30c42684848227c28f281c14d89d5c49e3cade0"} Oct 30 22:28:10.952252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:10.951952 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e3965ccb4e186a99eec7e6aa30c42684848227c28f281c14d89d5c49e3cade0" Oct 30 22:28:11.386321 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:11.386286 4335 patch_prober.go:29] interesting pod/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 22:28:11.386571 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:11.386336 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" podUID="c280505e-e32e-4a17-9643-3be7e1250e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 22:28:11.949156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:11.949117 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" podUID="9bbb21e7-621e-4490-bc0b-f31ee5b34404" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:12.386554 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.386518 4335 patch_prober.go:29] interesting pod/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 22:28:12.387010 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.386574 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" podUID="c280505e-e32e-4a17-9643-3be7e1250e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 22:28:12.483941 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.483752 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" containerID="cri-o://c28e2eef446723994ba4dbca5acc6555287ac7319b5cecd09e65dcc32fd18de2" gracePeriod=45 Oct 30 22:28:12.532640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.530155 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": dial tcp 10.133.0.81:8000: connect: connection refused" Oct 30 22:28:12.561876 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.561846 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l"] Oct 30 22:28:12.562816 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.562796 4335 cpu_manager.go:414] "containerMap: removing container" podUID="bd40b3cc-a54a-4d37-957e-40715570dbbf" containerName="revoke-keys" Oct 30 22:28:12.562942 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.562932 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd40b3cc-a54a-4d37-957e-40715570dbbf" containerName="revoke-keys" Oct 30 22:28:12.563144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.563135 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd40b3cc-a54a-4d37-957e-40715570dbbf" containerName="revoke-keys" Oct 30 22:28:12.566663 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.566637 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l" Oct 30 22:28:12.633637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.633353 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:28:12.633637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.633562 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:28:12.633637 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.633596 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:28:12.633950 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.633735 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:28:12.637697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.636269 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l"] Oct 30 22:28:12.761251 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.761075 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvn2w\" (UniqueName: \"kubernetes.io/projected/73b4c13f-c140-4d54-a269-317da161c851-kube-api-access-hvn2w\") pod \"maas-api-revoke-sub-0d5fc2b0-5727l\" (UID: \"73b4c13f-c140-4d54-a269-317da161c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l" Oct 30 22:28:12.761549 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.761492 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b4c13f-c140-4d54-a269-317da161c851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d5fc2b0-5727l\" (UID: \"73b4c13f-c140-4d54-a269-317da161c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l" Oct 30 22:28:12.761982 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.761930 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b4c13f-c140-4d54-a269-317da161c851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d5fc2b0-5727l\" (UID: \"73b4c13f-c140-4d54-a269-317da161c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l" Oct 30 22:28:12.863371 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.863337 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvn2w\" (UniqueName: \"kubernetes.io/projected/73b4c13f-c140-4d54-a269-317da161c851-kube-api-access-hvn2w\") pod \"maas-api-revoke-sub-0d5fc2b0-5727l\" (UID: \"73b4c13f-c140-4d54-a269-317da161c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l" Oct 30 22:28:12.863576 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.863416 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b4c13f-c140-4d54-a269-317da161c851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d5fc2b0-5727l\" (UID: \"73b4c13f-c140-4d54-a269-317da161c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l" Oct 30 22:28:12.863576 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.863440 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b4c13f-c140-4d54-a269-317da161c851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d5fc2b0-5727l\" (UID: \"73b4c13f-c140-4d54-a269-317da161c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l" Oct 30 22:28:12.864504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.864477 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b4c13f-c140-4d54-a269-317da161c851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d5fc2b0-5727l\" (UID: \"73b4c13f-c140-4d54-a269-317da161c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l" Oct 30 22:28:12.866014 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.865992 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b4c13f-c140-4d54-a269-317da161c851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d5fc2b0-5727l\" (UID: \"73b4c13f-c140-4d54-a269-317da161c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l" Oct 30 22:28:12.919523 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.919458 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:28:12.943662 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.941002 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvn2w\" (UniqueName: \"kubernetes.io/projected/73b4c13f-c140-4d54-a269-317da161c851-kube-api-access-hvn2w\") pod \"maas-api-revoke-sub-0d5fc2b0-5727l\" (UID: \"73b4c13f-c140-4d54-a269-317da161c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l" Oct 30 22:28:12.964006 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.963979 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181e8750-1ec6-4564-a9d3-390e703dce76-kube-api-access-cqjxw\" (UniqueName: \"kubernetes.io/projected/181e8750-1ec6-4564-a9d3-390e703dce76-kube-api-access-cqjxw\") pod \"181e8750-1ec6-4564-a9d3-390e703dce76\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " Oct 30 22:28:12.964157 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.964087 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-model-cache\") pod \"181e8750-1ec6-4564-a9d3-390e703dce76\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " Oct 30 22:28:12.964230 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.964162 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181e8750-1ec6-4564-a9d3-390e703dce76-tls-certs\" (UniqueName: \"kubernetes.io/secret/181e8750-1ec6-4564-a9d3-390e703dce76-tls-certs\") pod \"181e8750-1ec6-4564-a9d3-390e703dce76\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " Oct 30 22:28:12.964230 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.964201 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-home\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-home\") pod \"181e8750-1ec6-4564-a9d3-390e703dce76\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " Oct 30 22:28:12.964339 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.964266 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-dshm\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-dshm\") pod \"181e8750-1ec6-4564-a9d3-390e703dce76\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " Oct 30 22:28:12.964339 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.964308 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-tmp-dir\") pod \"181e8750-1ec6-4564-a9d3-390e703dce76\" (UID: \"181e8750-1ec6-4564-a9d3-390e703dce76\") " Oct 30 22:28:12.964791 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.964765 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-home" pod "181e8750-1ec6-4564-a9d3-390e703dce76" (UID: "181e8750-1ec6-4564-a9d3-390e703dce76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:12.965116 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.965097 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-model-cache" pod "181e8750-1ec6-4564-a9d3-390e703dce76" (UID: "181e8750-1ec6-4564-a9d3-390e703dce76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:12.965578 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.965548 4335 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-model-cache\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:12.965750 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.965735 4335 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-home\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:12.965847 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.965761 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-tmp-dir" pod "181e8750-1ec6-4564-a9d3-390e703dce76" (UID: "181e8750-1ec6-4564-a9d3-390e703dce76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:12.969112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.968137 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181e8750-1ec6-4564-a9d3-390e703dce76-kube-api-access-cqjxw" pod "181e8750-1ec6-4564-a9d3-390e703dce76" (UID: "181e8750-1ec6-4564-a9d3-390e703dce76"). InnerVolumeSpecName "kube-api-access-cqjxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:12.969112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.968527 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181e8750-1ec6-4564-a9d3-390e703dce76-tls-certs" pod "181e8750-1ec6-4564-a9d3-390e703dce76" (UID: "181e8750-1ec6-4564-a9d3-390e703dce76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:12.970426 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.970403 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-dshm" pod "181e8750-1ec6-4564-a9d3-390e703dce76" (UID: "181e8750-1ec6-4564-a9d3-390e703dce76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:12.972528 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.972505 4335 generic.go:356] "Generic (PLEG): container finished" podID="181e8750-1ec6-4564-a9d3-390e703dce76" containerID="c28e2eef446723994ba4dbca5acc6555287ac7319b5cecd09e65dcc32fd18de2" exitCode=0 Oct 30 22:28:12.972842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.972645 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" event={"ID":"181e8750-1ec6-4564-a9d3-390e703dce76","Type":"ContainerDied","Data":"c28e2eef446723994ba4dbca5acc6555287ac7319b5cecd09e65dcc32fd18de2"} Oct 30 22:28:12.972842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.972692 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" event={"ID":"181e8750-1ec6-4564-a9d3-390e703dce76","Type":"ContainerDied","Data":"644de6683fc4ecf3306fe3ab12db6800081cb67e7fd443ea9931cf4fcbefe975"} Oct 30 22:28:12.972842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.972714 4335 scope.go:122] "RemoveContainer" containerID="c28e2eef446723994ba4dbca5acc6555287ac7319b5cecd09e65dcc32fd18de2" Oct 30 22:28:12.972842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.972724 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh" Oct 30 22:28:12.993994 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.993960 4335 scope.go:122] "RemoveContainer" containerID="c28e2eef446723994ba4dbca5acc6555287ac7319b5cecd09e65dcc32fd18de2" Oct 30 22:28:12.994302 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:28:12.994275 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28e2eef446723994ba4dbca5acc6555287ac7319b5cecd09e65dcc32fd18de2\": container with ID starting with c28e2eef446723994ba4dbca5acc6555287ac7319b5cecd09e65dcc32fd18de2 not found: ID does not exist" containerID="c28e2eef446723994ba4dbca5acc6555287ac7319b5cecd09e65dcc32fd18de2" Oct 30 22:28:12.994406 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:12.994313 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28e2eef446723994ba4dbca5acc6555287ac7319b5cecd09e65dcc32fd18de2"} err="failed to get container status \"c28e2eef446723994ba4dbca5acc6555287ac7319b5cecd09e65dcc32fd18de2\": rpc error: code = NotFound desc = could not find container \"c28e2eef446723994ba4dbca5acc6555287ac7319b5cecd09e65dcc32fd18de2\": container with ID starting with c28e2eef446723994ba4dbca5acc6555287ac7319b5cecd09e65dcc32fd18de2 not found: ID does not exist" Oct 30 22:28:13.066552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:13.066523 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqjxw\" (UniqueName: \"kubernetes.io/projected/181e8750-1ec6-4564-a9d3-390e703dce76-kube-api-access-cqjxw\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:13.066552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:13.066548 4335 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/181e8750-1ec6-4564-a9d3-390e703dce76-tls-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:13.066552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:13.066559 4335 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-dshm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:13.066895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:13.066570 4335 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/181e8750-1ec6-4564-a9d3-390e703dce76-tmp-dir\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:13.158144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:13.158110 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh"] Oct 30 22:28:13.163460 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:13.163428 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7df73b/simulator-w0-7df73b-kserve-7fdddd8b5b-nvtjh"] Oct 30 22:28:13.192299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:13.192234 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l" Oct 30 22:28:13.386968 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:13.386935 4335 patch_prober.go:29] interesting pod/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 22:28:13.387377 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:13.386992 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" podUID="c280505e-e32e-4a17-9643-3be7e1250e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 22:28:13.636464 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:13.636363 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l"] Oct 30 22:28:13.643127 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:28:13.643096 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b4c13f_c140_4d54_a269_317da161c851.slice/crio-99d8e172f2100e97b90484f92db560259de475cb944bf2779872066fffed6b95 WatchSource:0}: Error finding container 99d8e172f2100e97b90484f92db560259de475cb944bf2779872066fffed6b95: Status 404 returned error can't find the container with id 99d8e172f2100e97b90484f92db560259de475cb944bf2779872066fffed6b95 Oct 30 22:28:13.982118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:13.982039 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l" event={"ID":"73b4c13f-c140-4d54-a269-317da161c851","Type":"ContainerStarted","Data":"99d8e172f2100e97b90484f92db560259de475cb944bf2779872066fffed6b95"} Oct 30 22:28:14.387045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:14.387009 4335 patch_prober.go:29] interesting pod/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 22:28:14.387504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:14.387069 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" podUID="c280505e-e32e-4a17-9643-3be7e1250e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 22:28:15.021244 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:15.021123 4335 generic.go:356] "Generic (PLEG): container finished" podID="73b4c13f-c140-4d54-a269-317da161c851" containerID="52afdd469c44597b336bd8a97d8584014a86f437b479fe04289297e020320d56" exitCode=0 Oct 30 22:28:15.021244 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:15.021210 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l" event={"ID":"73b4c13f-c140-4d54-a269-317da161c851","Type":"ContainerDied","Data":"52afdd469c44597b336bd8a97d8584014a86f437b479fe04289297e020320d56"} Oct 30 22:28:15.060243 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:15.060198 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" path="/var/lib/kubelet/pods/181e8750-1ec6-4564-a9d3-390e703dce76/volumes" Oct 30 22:28:15.390565 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:15.390539 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:28:15.391056 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:15.391041 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:28:15.391738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:15.391715 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:28:16.036158 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.036125 4335 generic.go:356] "Generic (PLEG): container finished" podID="ca1f40ea-c3a3-4522-b526-fb3aad6183fa" containerID="fde7e5a9aed94d157063a00d2e6bbe997f6916935068b72a84f500aa331eef6b" exitCode=0 Oct 30 22:28:16.036312 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.036198 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" event={"ID":"ca1f40ea-c3a3-4522-b526-fb3aad6183fa","Type":"ContainerDied","Data":"fde7e5a9aed94d157063a00d2e6bbe997f6916935068b72a84f500aa331eef6b"} Oct 30 22:28:16.109845 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.109815 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" Oct 30 22:28:16.199841 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.199806 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca1f40ea-c3a3-4522-b526-fb3aad6183fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1f40ea-c3a3-4522-b526-fb3aad6183fa-plugins-config-volume\") pod \"ca1f40ea-c3a3-4522-b526-fb3aad6183fa\" (UID: \"ca1f40ea-c3a3-4522-b526-fb3aad6183fa\") " Oct 30 22:28:16.200234 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.200071 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca1f40ea-c3a3-4522-b526-fb3aad6183fa-kube-api-access-6tzw8\" (UniqueName: \"kubernetes.io/projected/ca1f40ea-c3a3-4522-b526-fb3aad6183fa-kube-api-access-6tzw8\") pod \"ca1f40ea-c3a3-4522-b526-fb3aad6183fa\" (UID: \"ca1f40ea-c3a3-4522-b526-fb3aad6183fa\") " Oct 30 22:28:16.200570 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.200529 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca1f40ea-c3a3-4522-b526-fb3aad6183fa-plugins-config-volume" pod "ca1f40ea-c3a3-4522-b526-fb3aad6183fa" (UID: "ca1f40ea-c3a3-4522-b526-fb3aad6183fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:16.203234 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.203209 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca1f40ea-c3a3-4522-b526-fb3aad6183fa-kube-api-access-6tzw8" pod "ca1f40ea-c3a3-4522-b526-fb3aad6183fa" (UID: "ca1f40ea-c3a3-4522-b526-fb3aad6183fa"). InnerVolumeSpecName "kube-api-access-6tzw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:16.208711 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.208607 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l" Oct 30 22:28:16.257039 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.257016 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" Oct 30 22:28:16.300909 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.300886 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b4c13f-c140-4d54-a269-317da161c851-kube-api-access-hvn2w\" (UniqueName: \"kubernetes.io/projected/73b4c13f-c140-4d54-a269-317da161c851-kube-api-access-hvn2w\") pod \"73b4c13f-c140-4d54-a269-317da161c851\" (UID: \"73b4c13f-c140-4d54-a269-317da161c851\") " Oct 30 22:28:16.301013 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.300934 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bbb21e7-621e-4490-bc0b-f31ee5b34404-kube-api-access-2vjv7\" (UniqueName: \"kubernetes.io/projected/9bbb21e7-621e-4490-bc0b-f31ee5b34404-kube-api-access-2vjv7\") pod \"9bbb21e7-621e-4490-bc0b-f31ee5b34404\" (UID: \"9bbb21e7-621e-4490-bc0b-f31ee5b34404\") " Oct 30 22:28:16.301013 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.301006 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73b4c13f-c140-4d54-a269-317da161c851-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b4c13f-c140-4d54-a269-317da161c851-maas-api-openshift-ca\") pod \"73b4c13f-c140-4d54-a269-317da161c851\" (UID: \"73b4c13f-c140-4d54-a269-317da161c851\") " Oct 30 22:28:16.301152 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.301044 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73b4c13f-c140-4d54-a269-317da161c851-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b4c13f-c140-4d54-a269-317da161c851-maas-api-xks-ca\") pod \"73b4c13f-c140-4d54-a269-317da161c851\" (UID: \"73b4c13f-c140-4d54-a269-317da161c851\") " Oct 30 22:28:16.301152 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.301092 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bbb21e7-621e-4490-bc0b-f31ee5b34404-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbb21e7-621e-4490-bc0b-f31ee5b34404-plugins-config-volume\") pod \"9bbb21e7-621e-4490-bc0b-f31ee5b34404\" (UID: \"9bbb21e7-621e-4490-bc0b-f31ee5b34404\") " Oct 30 22:28:16.301394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.301365 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b4c13f-c140-4d54-a269-317da161c851-maas-api-openshift-ca" pod "73b4c13f-c140-4d54-a269-317da161c851" (UID: "73b4c13f-c140-4d54-a269-317da161c851"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:16.301579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.301554 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bbb21e7-621e-4490-bc0b-f31ee5b34404-plugins-config-volume" pod "9bbb21e7-621e-4490-bc0b-f31ee5b34404" (UID: "9bbb21e7-621e-4490-bc0b-f31ee5b34404"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:16.301732 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.301659 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b4c13f-c140-4d54-a269-317da161c851-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:16.301732 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.301680 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbb21e7-621e-4490-bc0b-f31ee5b34404-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:16.301732 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.301693 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tzw8\" (UniqueName: \"kubernetes.io/projected/ca1f40ea-c3a3-4522-b526-fb3aad6183fa-kube-api-access-6tzw8\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:16.301732 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.301702 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1f40ea-c3a3-4522-b526-fb3aad6183fa-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:16.303646 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.303599 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bbb21e7-621e-4490-bc0b-f31ee5b34404-kube-api-access-2vjv7" pod "9bbb21e7-621e-4490-bc0b-f31ee5b34404" (UID: "9bbb21e7-621e-4490-bc0b-f31ee5b34404"). InnerVolumeSpecName "kube-api-access-2vjv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:16.303743 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.303687 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73b4c13f-c140-4d54-a269-317da161c851-maas-api-xks-ca" pod "73b4c13f-c140-4d54-a269-317da161c851" (UID: "73b4c13f-c140-4d54-a269-317da161c851"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:16.303788 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.303736 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b4c13f-c140-4d54-a269-317da161c851-kube-api-access-hvn2w" pod "73b4c13f-c140-4d54-a269-317da161c851" (UID: "73b4c13f-c140-4d54-a269-317da161c851"). InnerVolumeSpecName "kube-api-access-hvn2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:16.402303 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.402244 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvn2w\" (UniqueName: \"kubernetes.io/projected/73b4c13f-c140-4d54-a269-317da161c851-kube-api-access-hvn2w\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:16.402303 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.402269 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vjv7\" (UniqueName: \"kubernetes.io/projected/9bbb21e7-621e-4490-bc0b-f31ee5b34404-kube-api-access-2vjv7\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:16.402303 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:16.402279 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b4c13f-c140-4d54-a269-317da161c851-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:17.045263 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.045230 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l" Oct 30 22:28:17.045263 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.045260 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l" event={"ID":"73b4c13f-c140-4d54-a269-317da161c851","Type":"ContainerDied","Data":"99d8e172f2100e97b90484f92db560259de475cb944bf2779872066fffed6b95"} Oct 30 22:28:17.045483 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.045307 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99d8e172f2100e97b90484f92db560259de475cb944bf2779872066fffed6b95" Oct 30 22:28:17.046747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.046721 4335 generic.go:356] "Generic (PLEG): container finished" podID="9bbb21e7-621e-4490-bc0b-f31ee5b34404" containerID="54f53254f640eb49e4801bdeb5dc98c0b1647485b077e8648873a3938eee7721" exitCode=0 Oct 30 22:28:17.046888 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.046762 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" Oct 30 22:28:17.046888 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.046802 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" event={"ID":"9bbb21e7-621e-4490-bc0b-f31ee5b34404","Type":"ContainerDied","Data":"54f53254f640eb49e4801bdeb5dc98c0b1647485b077e8648873a3938eee7721"} Oct 30 22:28:17.046888 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.046851 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q" event={"ID":"9bbb21e7-621e-4490-bc0b-f31ee5b34404","Type":"ContainerDied","Data":"a2e17639cc3ab420469a33ba5cb83cd7142412e7b8a08b7915107389ab215914"} Oct 30 22:28:17.046888 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.046881 4335 scope.go:122] "RemoveContainer" containerID="54f53254f640eb49e4801bdeb5dc98c0b1647485b077e8648873a3938eee7721" Oct 30 22:28:17.048693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.048648 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" event={"ID":"ca1f40ea-c3a3-4522-b526-fb3aad6183fa","Type":"ContainerDied","Data":"b1289d1f60574c2665c10d688b480a69c5a464e46cb0c1ee74071a98350a1e24"} Oct 30 22:28:17.048814 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.048693 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj" Oct 30 22:28:17.061099 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.061082 4335 scope.go:122] "RemoveContainer" containerID="54f53254f640eb49e4801bdeb5dc98c0b1647485b077e8648873a3938eee7721" Oct 30 22:28:17.061402 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:28:17.061379 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f53254f640eb49e4801bdeb5dc98c0b1647485b077e8648873a3938eee7721\": container with ID starting with 54f53254f640eb49e4801bdeb5dc98c0b1647485b077e8648873a3938eee7721 not found: ID does not exist" containerID="54f53254f640eb49e4801bdeb5dc98c0b1647485b077e8648873a3938eee7721" Oct 30 22:28:17.061497 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.061411 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f53254f640eb49e4801bdeb5dc98c0b1647485b077e8648873a3938eee7721"} err="failed to get container status \"54f53254f640eb49e4801bdeb5dc98c0b1647485b077e8648873a3938eee7721\": rpc error: code = NotFound desc = could not find container \"54f53254f640eb49e4801bdeb5dc98c0b1647485b077e8648873a3938eee7721\": container with ID starting with 54f53254f640eb49e4801bdeb5dc98c0b1647485b077e8648873a3938eee7721 not found: ID does not exist" Oct 30 22:28:17.061497 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.061433 4335 scope.go:122] "RemoveContainer" containerID="fde7e5a9aed94d157063a00d2e6bbe997f6916935068b72a84f500aa331eef6b" Oct 30 22:28:17.268039 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.267978 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q"] Oct 30 22:28:17.317080 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.317049 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b2f165-697c5bf8bc-9pt9q"] Oct 30 22:28:17.381725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.381698 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj"] Oct 30 22:28:17.387900 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.387877 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b2f165-fcb55dbdc-r7llj"] Oct 30 22:28:17.908155 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.908115 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" podUID="f85380bc-9459-4ef3-94c1-663683361092" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:17.908645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:17.908123 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" podUID="8a7601c3-d83d-4d7f-b5d3-7528cba61d67" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:19.057791 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:19.057760 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bbb21e7-621e-4490-bc0b-f31ee5b34404" path="/var/lib/kubelet/pods/9bbb21e7-621e-4490-bc0b-f31ee5b34404/volumes" Oct 30 22:28:19.058188 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:19.058139 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca1f40ea-c3a3-4522-b526-fb3aad6183fa" path="/var/lib/kubelet/pods/ca1f40ea-c3a3-4522-b526-fb3aad6183fa/volumes" Oct 30 22:28:25.321528 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:28:25.321491 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:28:27.909428 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:27.909386 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" podUID="8a7601c3-d83d-4d7f-b5d3-7528cba61d67" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:27.910394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:27.909461 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" podUID="f85380bc-9459-4ef3-94c1-663683361092" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:28.718888 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:28.718831 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" Oct 30 22:28:28.810297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:28.810257 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:28:28.843008 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:28.842985 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" Oct 30 22:28:28.860067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:28.860045 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a7601c3-d83d-4d7f-b5d3-7528cba61d67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7601c3-d83d-4d7f-b5d3-7528cba61d67-plugins-config-volume\") pod \"8a7601c3-d83d-4d7f-b5d3-7528cba61d67\" (UID: \"8a7601c3-d83d-4d7f-b5d3-7528cba61d67\") " Oct 30 22:28:28.860210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:28.860197 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7601c3-d83d-4d7f-b5d3-7528cba61d67-kube-api-access-lsnbt\" (UniqueName: \"kubernetes.io/projected/8a7601c3-d83d-4d7f-b5d3-7528cba61d67-kube-api-access-lsnbt\") pod \"8a7601c3-d83d-4d7f-b5d3-7528cba61d67\" (UID: \"8a7601c3-d83d-4d7f-b5d3-7528cba61d67\") " Oct 30 22:28:28.860456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:28.860433 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7601c3-d83d-4d7f-b5d3-7528cba61d67-plugins-config-volume" pod "8a7601c3-d83d-4d7f-b5d3-7528cba61d67" (UID: "8a7601c3-d83d-4d7f-b5d3-7528cba61d67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:28.860653 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:28.860636 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7601c3-d83d-4d7f-b5d3-7528cba61d67-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:28.862683 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:28.862664 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7601c3-d83d-4d7f-b5d3-7528cba61d67-kube-api-access-lsnbt" pod "8a7601c3-d83d-4d7f-b5d3-7528cba61d67" (UID: "8a7601c3-d83d-4d7f-b5d3-7528cba61d67"). InnerVolumeSpecName "kube-api-access-lsnbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:28.961340 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:28.961308 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f85380bc-9459-4ef3-94c1-663683361092-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85380bc-9459-4ef3-94c1-663683361092-plugins-config-volume\") pod \"f85380bc-9459-4ef3-94c1-663683361092\" (UID: \"f85380bc-9459-4ef3-94c1-663683361092\") " Oct 30 22:28:28.961846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:28.961388 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85380bc-9459-4ef3-94c1-663683361092-kube-api-access-j2qpp\" (UniqueName: \"kubernetes.io/projected/f85380bc-9459-4ef3-94c1-663683361092-kube-api-access-j2qpp\") pod \"f85380bc-9459-4ef3-94c1-663683361092\" (UID: \"f85380bc-9459-4ef3-94c1-663683361092\") " Oct 30 22:28:28.962063 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:28.962041 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsnbt\" (UniqueName: \"kubernetes.io/projected/8a7601c3-d83d-4d7f-b5d3-7528cba61d67-kube-api-access-lsnbt\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:28.962937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:28.962909 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f85380bc-9459-4ef3-94c1-663683361092-plugins-config-volume" pod "f85380bc-9459-4ef3-94c1-663683361092" (UID: "f85380bc-9459-4ef3-94c1-663683361092"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:28.965468 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:28.965446 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85380bc-9459-4ef3-94c1-663683361092-kube-api-access-j2qpp" pod "f85380bc-9459-4ef3-94c1-663683361092" (UID: "f85380bc-9459-4ef3-94c1-663683361092"). InnerVolumeSpecName "kube-api-access-j2qpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:29.063375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.063344 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85380bc-9459-4ef3-94c1-663683361092-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:29.063375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.063377 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2qpp\" (UniqueName: \"kubernetes.io/projected/f85380bc-9459-4ef3-94c1-663683361092-kube-api-access-j2qpp\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:29.195462 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.195424 4335 generic.go:356] "Generic (PLEG): container finished" podID="8a7601c3-d83d-4d7f-b5d3-7528cba61d67" containerID="db3836b80dd561fdf8585ab351e6e3a3d9211908021d520c7af2c91b8e8b740a" exitCode=0 Oct 30 22:28:29.195930 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.195498 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" Oct 30 22:28:29.196470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.195522 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" event={"ID":"8a7601c3-d83d-4d7f-b5d3-7528cba61d67","Type":"ContainerDied","Data":"db3836b80dd561fdf8585ab351e6e3a3d9211908021d520c7af2c91b8e8b740a"} Oct 30 22:28:29.196470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.196334 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn" event={"ID":"8a7601c3-d83d-4d7f-b5d3-7528cba61d67","Type":"ContainerDied","Data":"cfb567cbda392ac18caa1ed91e2df3d1de12a36cc7b500964177b73c84e00209"} Oct 30 22:28:29.196470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.196372 4335 scope.go:122] "RemoveContainer" containerID="db3836b80dd561fdf8585ab351e6e3a3d9211908021d520c7af2c91b8e8b740a" Oct 30 22:28:29.201114 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.201085 4335 generic.go:356] "Generic (PLEG): container finished" podID="f85380bc-9459-4ef3-94c1-663683361092" containerID="6c22a1475ab2a012cdf840909d2814a8ab103f4c6afb739adfc722cdb120ecba" exitCode=0 Oct 30 22:28:29.201237 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.201126 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" Oct 30 22:28:29.201237 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.201155 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" event={"ID":"f85380bc-9459-4ef3-94c1-663683361092","Type":"ContainerDied","Data":"6c22a1475ab2a012cdf840909d2814a8ab103f4c6afb739adfc722cdb120ecba"} Oct 30 22:28:29.201361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.201245 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j" event={"ID":"f85380bc-9459-4ef3-94c1-663683361092","Type":"ContainerDied","Data":"a0f87780893e6da22a73b64609ac73565a8958a36d4c305a4c2740856c0ebf16"} Oct 30 22:28:29.212502 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.212481 4335 scope.go:122] "RemoveContainer" containerID="db3836b80dd561fdf8585ab351e6e3a3d9211908021d520c7af2c91b8e8b740a" Oct 30 22:28:29.212824 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:28:29.212805 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db3836b80dd561fdf8585ab351e6e3a3d9211908021d520c7af2c91b8e8b740a\": container with ID starting with db3836b80dd561fdf8585ab351e6e3a3d9211908021d520c7af2c91b8e8b740a not found: ID does not exist" containerID="db3836b80dd561fdf8585ab351e6e3a3d9211908021d520c7af2c91b8e8b740a" Oct 30 22:28:29.212924 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.212835 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3836b80dd561fdf8585ab351e6e3a3d9211908021d520c7af2c91b8e8b740a"} err="failed to get container status \"db3836b80dd561fdf8585ab351e6e3a3d9211908021d520c7af2c91b8e8b740a\": rpc error: code = NotFound desc = could not find container \"db3836b80dd561fdf8585ab351e6e3a3d9211908021d520c7af2c91b8e8b740a\": container with ID starting with db3836b80dd561fdf8585ab351e6e3a3d9211908021d520c7af2c91b8e8b740a not found: ID does not exist" Oct 30 22:28:29.212924 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.212859 4335 scope.go:122] "RemoveContainer" containerID="6c22a1475ab2a012cdf840909d2814a8ab103f4c6afb739adfc722cdb120ecba" Oct 30 22:28:29.224429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.224412 4335 scope.go:122] "RemoveContainer" containerID="6c22a1475ab2a012cdf840909d2814a8ab103f4c6afb739adfc722cdb120ecba" Oct 30 22:28:29.224860 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:28:29.224839 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c22a1475ab2a012cdf840909d2814a8ab103f4c6afb739adfc722cdb120ecba\": container with ID starting with 6c22a1475ab2a012cdf840909d2814a8ab103f4c6afb739adfc722cdb120ecba not found: ID does not exist" containerID="6c22a1475ab2a012cdf840909d2814a8ab103f4c6afb739adfc722cdb120ecba" Oct 30 22:28:29.224951 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.224869 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c22a1475ab2a012cdf840909d2814a8ab103f4c6afb739adfc722cdb120ecba"} err="failed to get container status \"6c22a1475ab2a012cdf840909d2814a8ab103f4c6afb739adfc722cdb120ecba\": rpc error: code = NotFound desc = could not find container \"6c22a1475ab2a012cdf840909d2814a8ab103f4c6afb739adfc722cdb120ecba\": container with ID starting with 6c22a1475ab2a012cdf840909d2814a8ab103f4c6afb739adfc722cdb120ecba not found: ID does not exist" Oct 30 22:28:29.361646 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.361544 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn"] Oct 30 22:28:29.365821 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.365800 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7df73b-78f696f597-ghbpn"] Oct 30 22:28:29.478584 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.478553 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j"] Oct 30 22:28:29.485229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.485206 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7df73b-849dbb7fdc-8xf9j"] Oct 30 22:28:29.911708 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:29.911674 4335 scope.go:122] "RemoveContainer" containerID="a94c0abc4963b570dfd55bdc4c7d8e63a8cce5946da8b70f4ea867167ed2bdcb" Oct 30 22:28:31.068347 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:31.068304 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a7601c3-d83d-4d7f-b5d3-7528cba61d67" path="/var/lib/kubelet/pods/8a7601c3-d83d-4d7f-b5d3-7528cba61d67/volumes" Oct 30 22:28:31.068796 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:31.068781 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f85380bc-9459-4ef3-94c1-663683361092" path="/var/lib/kubelet/pods/f85380bc-9459-4ef3-94c1-663683361092/volumes" Oct 30 22:28:40.535562 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.535483 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x"] Oct 30 22:28:40.536120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.536098 4335 cpu_manager.go:414] "containerMap: removing container" podUID="9bbb21e7-621e-4490-bc0b-f31ee5b34404" containerName="payload-pre-processing" Oct 30 22:28:40.536210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.536123 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bbb21e7-621e-4490-bc0b-f31ee5b34404" containerName="payload-pre-processing" Oct 30 22:28:40.536210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.536143 4335 cpu_manager.go:414] "containerMap: removing container" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" Oct 30 22:28:40.536210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.536152 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" Oct 30 22:28:40.536210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.536173 4335 cpu_manager.go:414] "containerMap: removing container" podUID="8a7601c3-d83d-4d7f-b5d3-7528cba61d67" containerName="payload-processing" Oct 30 22:28:40.536210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.536180 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7601c3-d83d-4d7f-b5d3-7528cba61d67" containerName="payload-processing" Oct 30 22:28:40.536210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.536197 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f85380bc-9459-4ef3-94c1-663683361092" containerName="payload-pre-processing" Oct 30 22:28:40.536210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.536204 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85380bc-9459-4ef3-94c1-663683361092" containerName="payload-pre-processing" Oct 30 22:28:40.536480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.536236 4335 cpu_manager.go:414] "containerMap: removing container" podUID="ca1f40ea-c3a3-4522-b526-fb3aad6183fa" containerName="payload-processing" Oct 30 22:28:40.536480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.536242 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca1f40ea-c3a3-4522-b526-fb3aad6183fa" containerName="payload-processing" Oct 30 22:28:40.536480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.536254 4335 cpu_manager.go:414] "containerMap: removing container" podUID="73b4c13f-c140-4d54-a269-317da161c851" containerName="revoke-keys" Oct 30 22:28:40.536480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.536259 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b4c13f-c140-4d54-a269-317da161c851" containerName="revoke-keys" Oct 30 22:28:40.536480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.536420 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b4c13f-c140-4d54-a269-317da161c851" containerName="revoke-keys" Oct 30 22:28:40.536480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.536441 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bbb21e7-621e-4490-bc0b-f31ee5b34404" containerName="payload-pre-processing" Oct 30 22:28:40.536480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.536452 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="181e8750-1ec6-4564-a9d3-390e703dce76" containerName="main" Oct 30 22:28:40.536480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.536462 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7601c3-d83d-4d7f-b5d3-7528cba61d67" containerName="payload-processing" Oct 30 22:28:40.536480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.536473 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85380bc-9459-4ef3-94c1-663683361092" containerName="payload-pre-processing" Oct 30 22:28:40.536942 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.536487 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca1f40ea-c3a3-4522-b526-fb3aad6183fa" containerName="payload-processing" Oct 30 22:28:40.542409 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.542386 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.587381 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.587345 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-workload-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.587483 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.587391 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-credential-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.587534 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.587487 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istiod-ca-cert\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.587534 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.587508 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-workload-certs\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.587683 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.587656 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-data\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.587749 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.587706 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-envoy\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.587790 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.587760 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-podinfo\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.587823 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.587800 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-token\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.587823 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.587817 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjw2k\" (UniqueName: \"kubernetes.io/projected/599a4e4a-aff7-4cd8-acfa-209cb87ee424-kube-api-access-vjw2k\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.644014 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.643977 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b2f165-openshift-default-dockercfg-5fv84\"" Oct 30 22:28:40.649067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.649040 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x"] Oct 30 22:28:40.688869 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.688841 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istiod-ca-cert\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.688978 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.688872 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-workload-certs\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.688978 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.688934 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-data\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.688978 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.688972 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-envoy\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.689144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.689010 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-podinfo\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.689144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.689047 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-token\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.689144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.689073 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjw2k\" (UniqueName: \"kubernetes.io/projected/599a4e4a-aff7-4cd8-acfa-209cb87ee424-kube-api-access-vjw2k\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.689144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.689114 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-workload-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.689363 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.689149 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-credential-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.689363 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.689318 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-workload-certs\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.689456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.689379 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-data\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.689591 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.689570 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-credential-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.689663 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.689642 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-workload-socket\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.689713 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.689660 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istiod-ca-cert\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.691768 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.691743 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-envoy\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.692073 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.692055 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-podinfo\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.755137 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.755109 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-token\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.755884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.755861 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjw2k\" (UniqueName: \"kubernetes.io/projected/599a4e4a-aff7-4cd8-acfa-209cb87ee424-kube-api-access-vjw2k\") pod \"e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:40.856559 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:40.856530 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:41.297094 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:41.297060 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x"] Oct 30 22:28:41.298167 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:28:41.298141 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599a4e4a_aff7_4cd8_acfa_209cb87ee424.slice/crio-77e144a703aff1cb44d989fa7ba375e861d0f5c36aa2e4deeb6d927af2cbb215 WatchSource:0}: Error finding container 77e144a703aff1cb44d989fa7ba375e861d0f5c36aa2e4deeb6d927af2cbb215: Status 404 returned error can't find the container with id 77e144a703aff1cb44d989fa7ba375e861d0f5c36aa2e4deeb6d927af2cbb215 Oct 30 22:28:41.300996 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:41.300952 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:28:41.301103 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:41.301038 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:28:41.301103 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:41.301072 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:28:42.328295 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:42.328243 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" event={"ID":"599a4e4a-aff7-4cd8-acfa-209cb87ee424","Type":"ContainerStarted","Data":"89acdaa494e0f6c22f50273c9849507a870c2cf3e4955849c12a77e487828a00"} Oct 30 22:28:42.328975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:42.328319 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" event={"ID":"599a4e4a-aff7-4cd8-acfa-209cb87ee424","Type":"ContainerStarted","Data":"77e144a703aff1cb44d989fa7ba375e861d0f5c36aa2e4deeb6d927af2cbb215"} Oct 30 22:28:42.857112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:42.857083 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:42.858778 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:42.858754 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 22:28:42.858940 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:42.858819 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" podUID="599a4e4a-aff7-4cd8-acfa-209cb87ee424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 22:28:43.857141 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:43.857102 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 22:28:43.857529 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:43.857153 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" podUID="599a4e4a-aff7-4cd8-acfa-209cb87ee424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 22:28:44.857138 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:44.857104 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 22:28:44.857322 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:44.857166 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" podUID="599a4e4a-aff7-4cd8-acfa-209cb87ee424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 22:28:45.857877 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:45.857841 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 22:28:45.858284 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:45.857893 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" podUID="599a4e4a-aff7-4cd8-acfa-209cb87ee424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 22:28:46.857907 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:46.857871 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 22:28:46.858291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:46.857940 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" podUID="599a4e4a-aff7-4cd8-acfa-209cb87ee424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 22:28:47.856959 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:47.856924 4335 patch_prober.go:29] interesting pod/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 22:28:47.857228 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:47.856988 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" podUID="599a4e4a-aff7-4cd8-acfa-209cb87ee424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 22:28:48.861921 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:48.861892 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:48.862399 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:48.862372 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:48.863097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:48.863079 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:28:49.084927 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:49.084873 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" podStartSLOduration=9.08486082 podStartE2EDuration="9.08486082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:42.521569216 +0000 UTC m=+2655.622000954" watchObservedRunningTime="2026-10-30 22:28:49.08486082 +0000 UTC m=+2662.185292551" Oct 30 22:28:53.355158 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.355125 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt"] Oct 30 22:28:53.360479 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.360458 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" Oct 30 22:28:53.418173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.418143 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt"] Oct 30 22:28:53.448270 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.446283 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/138f5ad6-0075-4afe-abb6-526d5c89d7ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt\" (UID: \"138f5ad6-0075-4afe-abb6-526d5c89d7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" Oct 30 22:28:53.448270 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.446369 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57d7d\" (UniqueName: \"kubernetes.io/projected/138f5ad6-0075-4afe-abb6-526d5c89d7ad-kube-api-access-57d7d\") pod \"payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt\" (UID: \"138f5ad6-0075-4afe-abb6-526d5c89d7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" Oct 30 22:28:53.475957 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.475922 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c04440\"" Oct 30 22:28:53.477016 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.476326 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c04440-dockercfg-tcs2d\"" Oct 30 22:28:53.511726 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.500348 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7"] Oct 30 22:28:53.518944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.518879 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7" Oct 30 22:28:53.547530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.547500 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/138f5ad6-0075-4afe-abb6-526d5c89d7ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt\" (UID: \"138f5ad6-0075-4afe-abb6-526d5c89d7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" Oct 30 22:28:53.547682 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.547581 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57d7d\" (UniqueName: \"kubernetes.io/projected/138f5ad6-0075-4afe-abb6-526d5c89d7ad-kube-api-access-57d7d\") pod \"payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt\" (UID: \"138f5ad6-0075-4afe-abb6-526d5c89d7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" Oct 30 22:28:53.548373 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.548351 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/138f5ad6-0075-4afe-abb6-526d5c89d7ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt\" (UID: \"138f5ad6-0075-4afe-abb6-526d5c89d7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" Oct 30 22:28:53.561035 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.561009 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7"] Oct 30 22:28:53.609530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.609466 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57d7d\" (UniqueName: \"kubernetes.io/projected/138f5ad6-0075-4afe-abb6-526d5c89d7ad-kube-api-access-57d7d\") pod \"payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt\" (UID: \"138f5ad6-0075-4afe-abb6-526d5c89d7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" Oct 30 22:28:53.648681 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.648646 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32093b24-028d-4771-8aff-db94898573cd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7\" (UID: \"32093b24-028d-4771-8aff-db94898573cd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7" Oct 30 22:28:53.648817 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.648707 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9knb\" (UniqueName: \"kubernetes.io/projected/32093b24-028d-4771-8aff-db94898573cd-kube-api-access-d9knb\") pod \"payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7\" (UID: \"32093b24-028d-4771-8aff-db94898573cd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7" Oct 30 22:28:53.674998 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.674974 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" Oct 30 22:28:53.750591 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.750558 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32093b24-028d-4771-8aff-db94898573cd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7\" (UID: \"32093b24-028d-4771-8aff-db94898573cd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7" Oct 30 22:28:53.750773 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.750656 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9knb\" (UniqueName: \"kubernetes.io/projected/32093b24-028d-4771-8aff-db94898573cd-kube-api-access-d9knb\") pod \"payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7\" (UID: \"32093b24-028d-4771-8aff-db94898573cd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7" Oct 30 22:28:53.751725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.751698 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32093b24-028d-4771-8aff-db94898573cd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7\" (UID: \"32093b24-028d-4771-8aff-db94898573cd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7" Oct 30 22:28:53.813311 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.813279 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9knb\" (UniqueName: \"kubernetes.io/projected/32093b24-028d-4771-8aff-db94898573cd-kube-api-access-d9knb\") pod \"payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7\" (UID: \"32093b24-028d-4771-8aff-db94898573cd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7" Oct 30 22:28:53.842821 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:53.842799 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7" Oct 30 22:28:54.058941 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:54.058915 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt"] Oct 30 22:28:54.059981 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:28:54.059954 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138f5ad6_0075_4afe_abb6_526d5c89d7ad.slice/crio-3b75afc0efa9b16ee131d643afe4e7ab57e6070a8fe71e06adba50cb889d7cf4 WatchSource:0}: Error finding container 3b75afc0efa9b16ee131d643afe4e7ab57e6070a8fe71e06adba50cb889d7cf4: Status 404 returned error can't find the container with id 3b75afc0efa9b16ee131d643afe4e7ab57e6070a8fe71e06adba50cb889d7cf4 Oct 30 22:28:54.223918 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:54.223889 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7"] Oct 30 22:28:54.224827 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:28:54.224801 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32093b24_028d_4771_8aff_db94898573cd.slice/crio-d73cdd5616d24b44944c43bc8cef3e371e9a5ebd0c789eaa1c80b63bb7b0175b WatchSource:0}: Error finding container d73cdd5616d24b44944c43bc8cef3e371e9a5ebd0c789eaa1c80b63bb7b0175b: Status 404 returned error can't find the container with id d73cdd5616d24b44944c43bc8cef3e371e9a5ebd0c789eaa1c80b63bb7b0175b Oct 30 22:28:54.433526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:54.433265 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7" event={"ID":"32093b24-028d-4771-8aff-db94898573cd","Type":"ContainerStarted","Data":"dc8158f4620dde8a7a3654839434903c49930f4db77bce7d269ac878b797d1b7"} Oct 30 22:28:54.433526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:54.433327 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7" event={"ID":"32093b24-028d-4771-8aff-db94898573cd","Type":"ContainerStarted","Data":"d73cdd5616d24b44944c43bc8cef3e371e9a5ebd0c789eaa1c80b63bb7b0175b"} Oct 30 22:28:54.433526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:54.433493 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7" Oct 30 22:28:54.435210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:54.435181 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" event={"ID":"138f5ad6-0075-4afe-abb6-526d5c89d7ad","Type":"ContainerStarted","Data":"aae2c44a71468de9a2d3c8652184bd1676888f0e26e2ef2a8ffa2896255d3749"} Oct 30 22:28:54.435344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:54.435224 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" event={"ID":"138f5ad6-0075-4afe-abb6-526d5c89d7ad","Type":"ContainerStarted","Data":"3b75afc0efa9b16ee131d643afe4e7ab57e6070a8fe71e06adba50cb889d7cf4"} Oct 30 22:28:54.435344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:54.435253 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" Oct 30 22:28:54.551991 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:54.551883 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7" podStartSLOduration=1.551860827 podStartE2EDuration="1.551860827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:54.55079668 +0000 UTC m=+2667.651228413" watchObservedRunningTime="2026-10-30 22:28:54.551860827 +0000 UTC m=+2667.652292559" Oct 30 22:28:54.667222 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:54.667173 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" podStartSLOduration=1.66715956 podStartE2EDuration="1.66715956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:54.665728562 +0000 UTC m=+2667.766160294" watchObservedRunningTime="2026-10-30 22:28:54.66715956 +0000 UTC m=+2667.767591301" Oct 30 22:28:55.360119 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:28:55.360081 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:28:56.586750 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:56.586716 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp"] Oct 30 22:28:56.635688 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:56.635654 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7df73b-bqfhp"] Oct 30 22:28:57.060766 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:57.060733 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93708d6a-0a8e-4730-86ef-eb02f3eca95d" path="/var/lib/kubelet/pods/93708d6a-0a8e-4730-86ef-eb02f3eca95d/volumes" Oct 30 22:28:57.933001 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:57.932966 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf"] Oct 30 22:28:57.982028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:57.981995 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b2f165-m78jf"] Oct 30 22:28:59.057704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:28:59.057674 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e63d6602-8612-47cf-9174-a873eaf712b0" path="/var/lib/kubelet/pods/e63d6602-8612-47cf-9174-a873eaf712b0/volumes" Oct 30 22:29:02.501510 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.501475 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm"] Oct 30 22:29:02.507007 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.506983 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:02.562941 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.562910 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm"] Oct 30 22:29:02.607276 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.607249 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-026c11\"/\"openshift-service-ca.crt\"" Oct 30 22:29:02.607454 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.607436 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-026c11\"/\"kube-root-ca.crt\"" Oct 30 22:29:02.608869 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.608853 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-026c11\"/\"default-dockercfg-tpnsm\"" Oct 30 22:29:02.608968 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.608872 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-026c11\"/\"test-model-026c11-kserve-self-signed-certs\"" Oct 30 22:29:02.657301 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.657275 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjwj7\" (UniqueName: \"kubernetes.io/projected/b9e3ce96-8e7f-410f-b717-b6651336c373-kube-api-access-cjwj7\") pod \"test-model-026c11-kserve-5c6f4d5989-c88hm\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:02.657397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.657305 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e3ce96-8e7f-410f-b717-b6651336c373-tls-certs\") pod \"test-model-026c11-kserve-5c6f4d5989-c88hm\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:02.657397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.657347 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-tmp-dir\") pod \"test-model-026c11-kserve-5c6f4d5989-c88hm\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:02.657503 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.657485 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-home\") pod \"test-model-026c11-kserve-5c6f4d5989-c88hm\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:02.657548 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.657521 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-model-cache\") pod \"test-model-026c11-kserve-5c6f4d5989-c88hm\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:02.657548 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.657539 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-dshm\") pod \"test-model-026c11-kserve-5c6f4d5989-c88hm\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:02.758559 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.758507 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-home\") pod \"test-model-026c11-kserve-5c6f4d5989-c88hm\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:02.758559 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.758542 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-model-cache\") pod \"test-model-026c11-kserve-5c6f4d5989-c88hm\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:02.758745 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.758564 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-dshm\") pod \"test-model-026c11-kserve-5c6f4d5989-c88hm\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:02.758745 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.758660 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjwj7\" (UniqueName: \"kubernetes.io/projected/b9e3ce96-8e7f-410f-b717-b6651336c373-kube-api-access-cjwj7\") pod \"test-model-026c11-kserve-5c6f4d5989-c88hm\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:02.758745 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.758680 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e3ce96-8e7f-410f-b717-b6651336c373-tls-certs\") pod \"test-model-026c11-kserve-5c6f4d5989-c88hm\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:02.758745 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.758733 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-tmp-dir\") pod \"test-model-026c11-kserve-5c6f4d5989-c88hm\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:02.759028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.758985 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-model-cache\") pod \"test-model-026c11-kserve-5c6f4d5989-c88hm\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:02.759028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.758996 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-home\") pod \"test-model-026c11-kserve-5c6f4d5989-c88hm\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:02.759173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.759072 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-tmp-dir\") pod \"test-model-026c11-kserve-5c6f4d5989-c88hm\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:02.761708 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.761082 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-dshm\") pod \"test-model-026c11-kserve-5c6f4d5989-c88hm\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:02.761708 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.761598 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e3ce96-8e7f-410f-b717-b6651336c373-tls-certs\") pod \"test-model-026c11-kserve-5c6f4d5989-c88hm\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:02.822381 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:02.822359 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjwj7\" (UniqueName: \"kubernetes.io/projected/b9e3ce96-8e7f-410f-b717-b6651336c373-kube-api-access-cjwj7\") pod \"test-model-026c11-kserve-5c6f4d5989-c88hm\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:03.120989 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:03.120960 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:03.503482 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:03.503451 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm"] Oct 30 22:29:03.504041 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:29:03.504018 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9e3ce96_8e7f_410f_b717_b6651336c373.slice/crio-97cec7493da509a54ceb0381d8d64580b2b7bfd8d9e4c4012690a7d2ee431f75 WatchSource:0}: Error finding container 97cec7493da509a54ceb0381d8d64580b2b7bfd8d9e4c4012690a7d2ee431f75: Status 404 returned error can't find the container with id 97cec7493da509a54ceb0381d8d64580b2b7bfd8d9e4c4012690a7d2ee431f75 Oct 30 22:29:04.451400 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:04.451368 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv"] Oct 30 22:29:04.451663 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:04.451602 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" podUID="6c2f07ba-2e3c-45b2-b619-7ce607083590" containerName="istio-proxy" containerID="cri-o://c18d86f1d774c013b51aa4ce8f4b3490e0029ebabc8e430e29da1f04aa3fdaeb" gracePeriod=30 Oct 30 22:29:04.545445 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:04.545404 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" event={"ID":"b9e3ce96-8e7f-410f-b717-b6651336c373","Type":"ContainerStarted","Data":"ccacbf1b2dcdf9f7e68b5cfd9332eb288ea126d3d731b04bc44f80d2e5f777e6"} Oct 30 22:29:04.550438 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:04.550386 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" event={"ID":"b9e3ce96-8e7f-410f-b717-b6651336c373","Type":"ContainerStarted","Data":"97cec7493da509a54ceb0381d8d64580b2b7bfd8d9e4c4012690a7d2ee431f75"} Oct 30 22:29:04.761933 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:04.761832 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" podStartSLOduration=2.761818867 podStartE2EDuration="2.761818867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:04.758887067 +0000 UTC m=+2677.859318800" watchObservedRunningTime="2026-10-30 22:29:04.761818867 +0000 UTC m=+2677.862250645" Oct 30 22:29:05.121976 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:05.121945 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:05.446832 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:05.446754 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" Oct 30 22:29:05.446832 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:05.446819 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7" Oct 30 22:29:05.617475 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:05.617445 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x"] Oct 30 22:29:05.617952 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:05.617656 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" podUID="599a4e4a-aff7-4cd8-acfa-209cb87ee424" containerName="istio-proxy" containerID="cri-o://89acdaa494e0f6c22f50273c9849507a870c2cf3e4955849c12a77e487828a00" gracePeriod=30 Oct 30 22:29:06.570936 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:06.570900 4335 patch_prober.go:29] interesting pod/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:29:06.571116 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:06.570963 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" podUID="6c2f07ba-2e3c-45b2-b619-7ce607083590" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:29:09.588971 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.588941 4335 generic.go:356] "Generic (PLEG): container finished" podID="6c2f07ba-2e3c-45b2-b619-7ce607083590" containerID="c18d86f1d774c013b51aa4ce8f4b3490e0029ebabc8e430e29da1f04aa3fdaeb" exitCode=0 Oct 30 22:29:09.589308 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.589013 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" event={"ID":"6c2f07ba-2e3c-45b2-b619-7ce607083590","Type":"ContainerDied","Data":"c18d86f1d774c013b51aa4ce8f4b3490e0029ebabc8e430e29da1f04aa3fdaeb"} Oct 30 22:29:09.724887 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.724836 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:29:09.842128 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.842101 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2f07ba-2e3c-45b2-b619-7ce607083590-kube-api-access-wqfmf\" (UniqueName: \"kubernetes.io/projected/6c2f07ba-2e3c-45b2-b619-7ce607083590-kube-api-access-wqfmf\") pod \"6c2f07ba-2e3c-45b2-b619-7ce607083590\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " Oct 30 22:29:09.842128 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.842139 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-podinfo\") pod \"6c2f07ba-2e3c-45b2-b619-7ce607083590\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " Oct 30 22:29:09.842343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.842173 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-token\" (UniqueName: \"kubernetes.io/projected/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-token\") pod \"6c2f07ba-2e3c-45b2-b619-7ce607083590\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " Oct 30 22:29:09.842343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.842217 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-data\") pod \"6c2f07ba-2e3c-45b2-b619-7ce607083590\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " Oct 30 22:29:09.842343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.842242 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-envoy\") pod \"6c2f07ba-2e3c-45b2-b619-7ce607083590\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " Oct 30 22:29:09.842343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.842302 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-workload-certs\") pod \"6c2f07ba-2e3c-45b2-b619-7ce607083590\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " Oct 30 22:29:09.842574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.842412 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-workload-socket\") pod \"6c2f07ba-2e3c-45b2-b619-7ce607083590\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " Oct 30 22:29:09.842574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.842454 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c2f07ba-2e3c-45b2-b619-7ce607083590-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2f07ba-2e3c-45b2-b619-7ce607083590-istiod-ca-cert\") pod \"6c2f07ba-2e3c-45b2-b619-7ce607083590\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " Oct 30 22:29:09.842574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.842510 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-credential-socket\") pod \"6c2f07ba-2e3c-45b2-b619-7ce607083590\" (UID: \"6c2f07ba-2e3c-45b2-b619-7ce607083590\") " Oct 30 22:29:09.842574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.842519 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-data" pod "6c2f07ba-2e3c-45b2-b619-7ce607083590" (UID: "6c2f07ba-2e3c-45b2-b619-7ce607083590"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:09.842845 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.842689 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-workload-socket" pod "6c2f07ba-2e3c-45b2-b619-7ce607083590" (UID: "6c2f07ba-2e3c-45b2-b619-7ce607083590"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:09.842845 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.842699 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-workload-certs" pod "6c2f07ba-2e3c-45b2-b619-7ce607083590" (UID: "6c2f07ba-2e3c-45b2-b619-7ce607083590"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:09.842967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.842847 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-credential-socket" pod "6c2f07ba-2e3c-45b2-b619-7ce607083590" (UID: "6c2f07ba-2e3c-45b2-b619-7ce607083590"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:09.842967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.842904 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c2f07ba-2e3c-45b2-b619-7ce607083590-istiod-ca-cert" pod "6c2f07ba-2e3c-45b2-b619-7ce607083590" (UID: "6c2f07ba-2e3c-45b2-b619-7ce607083590"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:09.843250 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.843024 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:09.843250 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.843044 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:09.843250 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.843059 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:09.843250 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.843076 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2f07ba-2e3c-45b2-b619-7ce607083590-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:09.843250 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.843089 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:09.844933 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.844904 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-token" pod "6c2f07ba-2e3c-45b2-b619-7ce607083590" (UID: "6c2f07ba-2e3c-45b2-b619-7ce607083590"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:09.845226 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.845208 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-envoy" pod "6c2f07ba-2e3c-45b2-b619-7ce607083590" (UID: "6c2f07ba-2e3c-45b2-b619-7ce607083590"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:09.845375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.845352 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2f07ba-2e3c-45b2-b619-7ce607083590-kube-api-access-wqfmf" pod "6c2f07ba-2e3c-45b2-b619-7ce607083590" (UID: "6c2f07ba-2e3c-45b2-b619-7ce607083590"). InnerVolumeSpecName "kube-api-access-wqfmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:09.845475 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.845376 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-podinfo" pod "6c2f07ba-2e3c-45b2-b619-7ce607083590" (UID: "6c2f07ba-2e3c-45b2-b619-7ce607083590"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:29:09.944791 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.944761 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqfmf\" (UniqueName: \"kubernetes.io/projected/6c2f07ba-2e3c-45b2-b619-7ce607083590-kube-api-access-wqfmf\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:09.944791 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.944789 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:09.944791 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.944800 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:09.945098 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:09.944809 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2f07ba-2e3c-45b2-b619-7ce607083590-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:10.598772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:10.598734 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" Oct 30 22:29:10.598772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:10.598743 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv" event={"ID":"6c2f07ba-2e3c-45b2-b619-7ce607083590","Type":"ContainerDied","Data":"3873de063c980302910ad46cfadd65aac7daedb71d55fcadb1c93b3b3eac47ac"} Oct 30 22:29:10.599275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:10.598820 4335 scope.go:122] "RemoveContainer" containerID="c18d86f1d774c013b51aa4ce8f4b3490e0029ebabc8e430e29da1f04aa3fdaeb" Oct 30 22:29:10.817199 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:10.817165 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv"] Oct 30 22:29:10.863650 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:10.863597 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7df73b-openshift-default-7fd847d4b7-grbsv"] Oct 30 22:29:10.881367 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:10.881347 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:29:11.056714 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.056680 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-data\") pod \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " Oct 30 22:29:11.056904 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.056720 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-workload-socket\") pod \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " Oct 30 22:29:11.056904 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.056746 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-envoy\") pod \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " Oct 30 22:29:11.056904 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.056774 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-workload-certs\") pod \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " Oct 30 22:29:11.056904 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.056830 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-credential-socket\") pod \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " Oct 30 22:29:11.056904 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.056854 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istiod-ca-cert\") pod \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " Oct 30 22:29:11.057172 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.056905 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-podinfo\") pod \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " Oct 30 22:29:11.057172 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.056944 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/599a4e4a-aff7-4cd8-acfa-209cb87ee424-kube-api-access-vjw2k\" (UniqueName: \"kubernetes.io/projected/599a4e4a-aff7-4cd8-acfa-209cb87ee424-kube-api-access-vjw2k\") pod \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " Oct 30 22:29:11.057172 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.056982 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-token\" (UniqueName: \"kubernetes.io/projected/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-token\") pod \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\" (UID: \"599a4e4a-aff7-4cd8-acfa-209cb87ee424\") " Oct 30 22:29:11.057172 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.056983 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-data" pod "599a4e4a-aff7-4cd8-acfa-209cb87ee424" (UID: "599a4e4a-aff7-4cd8-acfa-209cb87ee424"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:11.057172 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.057027 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-workload-socket" pod "599a4e4a-aff7-4cd8-acfa-209cb87ee424" (UID: "599a4e4a-aff7-4cd8-acfa-209cb87ee424"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:11.057431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.057298 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istiod-ca-cert" pod "599a4e4a-aff7-4cd8-acfa-209cb87ee424" (UID: "599a4e4a-aff7-4cd8-acfa-209cb87ee424"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:11.057431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.057300 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-credential-socket" pod "599a4e4a-aff7-4cd8-acfa-209cb87ee424" (UID: "599a4e4a-aff7-4cd8-acfa-209cb87ee424"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:11.057431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.057415 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-workload-certs" pod "599a4e4a-aff7-4cd8-acfa-209cb87ee424" (UID: "599a4e4a-aff7-4cd8-acfa-209cb87ee424"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:11.057635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.057571 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:11.057635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.057589 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:11.057635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.057603 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:11.057819 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.057637 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:11.057819 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.057652 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:11.058301 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.058278 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c2f07ba-2e3c-45b2-b619-7ce607083590" path="/var/lib/kubelet/pods/6c2f07ba-2e3c-45b2-b619-7ce607083590/volumes" Oct 30 22:29:11.059320 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.059299 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599a4e4a-aff7-4cd8-acfa-209cb87ee424-kube-api-access-vjw2k" pod "599a4e4a-aff7-4cd8-acfa-209cb87ee424" (UID: "599a4e4a-aff7-4cd8-acfa-209cb87ee424"). InnerVolumeSpecName "kube-api-access-vjw2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:11.059508 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.059493 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-token" pod "599a4e4a-aff7-4cd8-acfa-209cb87ee424" (UID: "599a4e4a-aff7-4cd8-acfa-209cb87ee424"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:11.059658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.059601 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-podinfo" pod "599a4e4a-aff7-4cd8-acfa-209cb87ee424" (UID: "599a4e4a-aff7-4cd8-acfa-209cb87ee424"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:29:11.060207 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.060187 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-envoy" pod "599a4e4a-aff7-4cd8-acfa-209cb87ee424" (UID: "599a4e4a-aff7-4cd8-acfa-209cb87ee424"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:11.158415 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.158359 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:11.158415 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.158384 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:11.158415 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.158393 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjw2k\" (UniqueName: \"kubernetes.io/projected/599a4e4a-aff7-4cd8-acfa-209cb87ee424-kube-api-access-vjw2k\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:11.158415 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.158402 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599a4e4a-aff7-4cd8-acfa-209cb87ee424-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:11.610285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.610250 4335 generic.go:356] "Generic (PLEG): container finished" podID="599a4e4a-aff7-4cd8-acfa-209cb87ee424" containerID="89acdaa494e0f6c22f50273c9849507a870c2cf3e4955849c12a77e487828a00" exitCode=0 Oct 30 22:29:11.610762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.610312 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" Oct 30 22:29:11.610762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.610339 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" event={"ID":"599a4e4a-aff7-4cd8-acfa-209cb87ee424","Type":"ContainerDied","Data":"89acdaa494e0f6c22f50273c9849507a870c2cf3e4955849c12a77e487828a00"} Oct 30 22:29:11.610762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.610393 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x" event={"ID":"599a4e4a-aff7-4cd8-acfa-209cb87ee424","Type":"ContainerDied","Data":"77e144a703aff1cb44d989fa7ba375e861d0f5c36aa2e4deeb6d927af2cbb215"} Oct 30 22:29:11.610762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.610417 4335 scope.go:122] "RemoveContainer" containerID="89acdaa494e0f6c22f50273c9849507a870c2cf3e4955849c12a77e487828a00" Oct 30 22:29:11.621918 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.621898 4335 scope.go:122] "RemoveContainer" containerID="89acdaa494e0f6c22f50273c9849507a870c2cf3e4955849c12a77e487828a00" Oct 30 22:29:11.622204 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:29:11.622185 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89acdaa494e0f6c22f50273c9849507a870c2cf3e4955849c12a77e487828a00\": container with ID starting with 89acdaa494e0f6c22f50273c9849507a870c2cf3e4955849c12a77e487828a00 not found: ID does not exist" containerID="89acdaa494e0f6c22f50273c9849507a870c2cf3e4955849c12a77e487828a00" Oct 30 22:29:11.622274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.622215 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89acdaa494e0f6c22f50273c9849507a870c2cf3e4955849c12a77e487828a00"} err="failed to get container status \"89acdaa494e0f6c22f50273c9849507a870c2cf3e4955849c12a77e487828a00\": rpc error: code = NotFound desc = could not find container \"89acdaa494e0f6c22f50273c9849507a870c2cf3e4955849c12a77e487828a00\": container with ID starting with 89acdaa494e0f6c22f50273c9849507a870c2cf3e4955849c12a77e487828a00 not found: ID does not exist" Oct 30 22:29:11.778024 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.777985 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x"] Oct 30 22:29:11.829336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:11.829302 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b2f165-openshift-default-6549966f6f-x6q9x"] Oct 30 22:29:12.578584 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:12.578550 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:29:13.058729 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:13.058698 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="599a4e4a-aff7-4cd8-acfa-209cb87ee424" path="/var/lib/kubelet/pods/599a4e4a-aff7-4cd8-acfa-209cb87ee424/volumes" Oct 30 22:29:13.121465 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:13.121436 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:13.134782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:13.134755 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:13.639176 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:13.639141 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:29:25.322889 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:29:25.322854 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:29:28.425846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:28.425815 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:29:28.504832 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:28.504802 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:29:29.983504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:29.983466 4335 scope.go:122] "RemoveContainer" containerID="e08c332d62cf3aa59bd9ec5d0b82caaea0abf022fc9aa8b23885b0ba3d9fc321" Oct 30 22:29:29.994082 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:29.994064 4335 scope.go:122] "RemoveContainer" containerID="ea858b00cd099926dc7c61b6c79579c04944d7f4e13527f55c4b726ed60c77dd" Oct 30 22:29:30.004646 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:30.004605 4335 scope.go:122] "RemoveContainer" containerID="e19633ccc8f4f5136721bfe2209f143841e1940be5e332154e457975de669a40" Oct 30 22:29:30.014173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:30.014157 4335 scope.go:122] "RemoveContainer" containerID="8b3c45f84291e0d9a59c7e4dd5e475000c6903c746dd7db8e424ae69524f59a7" Oct 30 22:29:40.507840 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:40.507801 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:29:55.057825 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:55.057794 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw"] Oct 30 22:29:55.100835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:55.100805 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d22538ad-8g2jw"] Oct 30 22:29:55.323124 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:29:55.323037 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:29:56.050031 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:56.049998 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7"] Oct 30 22:29:56.100029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:56.099970 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb3c91f-8xsj7"] Oct 30 22:29:57.058407 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:57.058372 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d41c004-bc10-4167-b242-cd963585abbe" path="/var/lib/kubelet/pods/1d41c004-bc10-4167-b242-cd963585abbe/volumes" Oct 30 22:29:57.058929 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:29:57.058907 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="374b8c7b-ddb6-4209-b099-4cfa35b82a2e" path="/var/lib/kubelet/pods/374b8c7b-ddb6-4209-b099-4cfa35b82a2e/volumes" Oct 30 22:30:00.174066 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.174026 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq"] Oct 30 22:30:00.174587 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.174542 4335 cpu_manager.go:414] "containerMap: removing container" podUID="6c2f07ba-2e3c-45b2-b619-7ce607083590" containerName="istio-proxy" Oct 30 22:30:00.174587 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.174553 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c2f07ba-2e3c-45b2-b619-7ce607083590" containerName="istio-proxy" Oct 30 22:30:00.174587 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.174572 4335 cpu_manager.go:414] "containerMap: removing container" podUID="599a4e4a-aff7-4cd8-acfa-209cb87ee424" containerName="istio-proxy" Oct 30 22:30:00.174587 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.174577 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="599a4e4a-aff7-4cd8-acfa-209cb87ee424" containerName="istio-proxy" Oct 30 22:30:00.174837 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.174696 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c2f07ba-2e3c-45b2-b619-7ce607083590" containerName="istio-proxy" Oct 30 22:30:00.174837 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.174706 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="599a4e4a-aff7-4cd8-acfa-209cb87ee424" containerName="istio-proxy" Oct 30 22:30:00.178300 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.178279 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq" Oct 30 22:30:00.191492 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.191468 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq"] Oct 30 22:30:00.202164 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.202139 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm"] Oct 30 22:30:00.202285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.202247 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" Oct 30 22:30:00.206109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.206091 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" Oct 30 22:30:00.223589 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.223565 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp"] Oct 30 22:30:00.227355 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.227337 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" Oct 30 22:30:00.236278 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.236261 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:30:00.238130 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.238111 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5"] Oct 30 22:30:00.241649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.241631 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" Oct 30 22:30:00.255848 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.255826 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x67bg\" (UniqueName: \"kubernetes.io/projected/5622d889-2cbb-4d12-8723-2211b3e79d35-kube-api-access-x67bg\") pod \"maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq\" (UID: \"5622d889-2cbb-4d12-8723-2211b3e79d35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq" Oct 30 22:30:00.255940 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.255881 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/151b7632-44e1-4a13-a598-e454d11ef480-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm\" (UID: \"151b7632-44e1-4a13-a598-e454d11ef480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" Oct 30 22:30:00.255990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.255970 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vph95\" (UniqueName: \"kubernetes.io/projected/ffd1f7df-e348-4601-bfa9-18b26ef818cc-kube-api-access-vph95\") pod \"maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq\" (UID: \"ffd1f7df-e348-4601-bfa9-18b26ef818cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" Oct 30 22:30:00.256042 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.256027 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzhd4\" (UniqueName: \"kubernetes.io/projected/151b7632-44e1-4a13-a598-e454d11ef480-kube-api-access-pzhd4\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm\" (UID: \"151b7632-44e1-4a13-a598-e454d11ef480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" Oct 30 22:30:00.256083 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.256062 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dnlj\" (UniqueName: \"kubernetes.io/projected/b80c8083-c781-47e2-ad75-9d2d7ab426b3-kube-api-access-8dnlj\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp\" (UID: \"b80c8083-c781-47e2-ad75-9d2d7ab426b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" Oct 30 22:30:00.256118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.256112 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5622d889-2cbb-4d12-8723-2211b3e79d35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq\" (UID: \"5622d889-2cbb-4d12-8723-2211b3e79d35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq" Oct 30 22:30:00.256210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.256198 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd1f7df-e348-4601-bfa9-18b26ef818cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq\" (UID: \"ffd1f7df-e348-4601-bfa9-18b26ef818cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" Oct 30 22:30:00.256293 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.256275 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc1395f-6452-49d7-9452-272f05d50de1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5\" (UID: \"bdc1395f-6452-49d7-9452-272f05d50de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" Oct 30 22:30:00.256332 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.256303 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b80c8083-c781-47e2-ad75-9d2d7ab426b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp\" (UID: \"b80c8083-c781-47e2-ad75-9d2d7ab426b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" Oct 30 22:30:00.256332 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.256324 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf7xp\" (UniqueName: \"kubernetes.io/projected/bdc1395f-6452-49d7-9452-272f05d50de1-kube-api-access-xf7xp\") pod \"maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5\" (UID: \"bdc1395f-6452-49d7-9452-272f05d50de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" Oct 30 22:30:00.269348 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.269323 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:30:00.269438 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.269354 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:30:00.280756 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.280737 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq"] Oct 30 22:30:00.287724 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.287701 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq"] Oct 30 22:30:00.335817 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.335789 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm"] Oct 30 22:30:00.339430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.339407 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp"] Oct 30 22:30:00.357709 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.357685 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd1f7df-e348-4601-bfa9-18b26ef818cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq\" (UID: \"ffd1f7df-e348-4601-bfa9-18b26ef818cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" Oct 30 22:30:00.357833 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.357735 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc1395f-6452-49d7-9452-272f05d50de1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5\" (UID: \"bdc1395f-6452-49d7-9452-272f05d50de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" Oct 30 22:30:00.357833 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.357752 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b80c8083-c781-47e2-ad75-9d2d7ab426b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp\" (UID: \"b80c8083-c781-47e2-ad75-9d2d7ab426b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" Oct 30 22:30:00.357833 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.357774 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf7xp\" (UniqueName: \"kubernetes.io/projected/bdc1395f-6452-49d7-9452-272f05d50de1-kube-api-access-xf7xp\") pod \"maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5\" (UID: \"bdc1395f-6452-49d7-9452-272f05d50de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" Oct 30 22:30:00.357833 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.357823 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x67bg\" (UniqueName: \"kubernetes.io/projected/5622d889-2cbb-4d12-8723-2211b3e79d35-kube-api-access-x67bg\") pod \"maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq\" (UID: \"5622d889-2cbb-4d12-8723-2211b3e79d35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq" Oct 30 22:30:00.358065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.357876 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/151b7632-44e1-4a13-a598-e454d11ef480-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm\" (UID: \"151b7632-44e1-4a13-a598-e454d11ef480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" Oct 30 22:30:00.358065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.357907 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vph95\" (UniqueName: \"kubernetes.io/projected/ffd1f7df-e348-4601-bfa9-18b26ef818cc-kube-api-access-vph95\") pod \"maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq\" (UID: \"ffd1f7df-e348-4601-bfa9-18b26ef818cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" Oct 30 22:30:00.358065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.357938 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzhd4\" (UniqueName: \"kubernetes.io/projected/151b7632-44e1-4a13-a598-e454d11ef480-kube-api-access-pzhd4\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm\" (UID: \"151b7632-44e1-4a13-a598-e454d11ef480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" Oct 30 22:30:00.358065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.357979 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dnlj\" (UniqueName: \"kubernetes.io/projected/b80c8083-c781-47e2-ad75-9d2d7ab426b3-kube-api-access-8dnlj\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp\" (UID: \"b80c8083-c781-47e2-ad75-9d2d7ab426b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" Oct 30 22:30:00.358065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.358019 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5622d889-2cbb-4d12-8723-2211b3e79d35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq\" (UID: \"5622d889-2cbb-4d12-8723-2211b3e79d35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq" Oct 30 22:30:00.358510 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.358486 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc1395f-6452-49d7-9452-272f05d50de1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5\" (UID: \"bdc1395f-6452-49d7-9452-272f05d50de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" Oct 30 22:30:00.358592 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.358570 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b80c8083-c781-47e2-ad75-9d2d7ab426b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp\" (UID: \"b80c8083-c781-47e2-ad75-9d2d7ab426b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" Oct 30 22:30:00.358724 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.358703 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd1f7df-e348-4601-bfa9-18b26ef818cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq\" (UID: \"ffd1f7df-e348-4601-bfa9-18b26ef818cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" Oct 30 22:30:00.358724 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.358717 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5622d889-2cbb-4d12-8723-2211b3e79d35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq\" (UID: \"5622d889-2cbb-4d12-8723-2211b3e79d35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq" Oct 30 22:30:00.359048 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.359031 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/151b7632-44e1-4a13-a598-e454d11ef480-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm\" (UID: \"151b7632-44e1-4a13-a598-e454d11ef480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" Oct 30 22:30:00.374605 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.374581 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5"] Oct 30 22:30:00.420544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.420518 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf7xp\" (UniqueName: \"kubernetes.io/projected/bdc1395f-6452-49d7-9452-272f05d50de1-kube-api-access-xf7xp\") pod \"maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5\" (UID: \"bdc1395f-6452-49d7-9452-272f05d50de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" Oct 30 22:30:00.420789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.420753 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dnlj\" (UniqueName: \"kubernetes.io/projected/b80c8083-c781-47e2-ad75-9d2d7ab426b3-kube-api-access-8dnlj\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp\" (UID: \"b80c8083-c781-47e2-ad75-9d2d7ab426b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" Oct 30 22:30:00.420912 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.420889 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x67bg\" (UniqueName: \"kubernetes.io/projected/5622d889-2cbb-4d12-8723-2211b3e79d35-kube-api-access-x67bg\") pod \"maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq\" (UID: \"5622d889-2cbb-4d12-8723-2211b3e79d35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq" Oct 30 22:30:00.421268 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.421247 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vph95\" (UniqueName: \"kubernetes.io/projected/ffd1f7df-e348-4601-bfa9-18b26ef818cc-kube-api-access-vph95\") pod \"maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq\" (UID: \"ffd1f7df-e348-4601-bfa9-18b26ef818cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" Oct 30 22:30:00.421701 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.421681 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzhd4\" (UniqueName: \"kubernetes.io/projected/151b7632-44e1-4a13-a598-e454d11ef480-kube-api-access-pzhd4\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm\" (UID: \"151b7632-44e1-4a13-a598-e454d11ef480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" Oct 30 22:30:00.503275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.503222 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq" Oct 30 22:30:00.518179 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.518153 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" Oct 30 22:30:00.524166 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.524144 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" Oct 30 22:30:00.541511 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.541484 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" Oct 30 22:30:00.553850 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.553825 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" Oct 30 22:30:00.901245 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:30:00.901204 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5622d889_2cbb_4d12_8723_2211b3e79d35.slice/crio-94d3e45ccd772fa8e1f074002e863a6558171449357a3e7ba32aa04bdca096e1 WatchSource:0}: Error finding container 94d3e45ccd772fa8e1f074002e863a6558171449357a3e7ba32aa04bdca096e1: Status 404 returned error can't find the container with id 94d3e45ccd772fa8e1f074002e863a6558171449357a3e7ba32aa04bdca096e1 Oct 30 22:30:00.901918 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.901892 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq"] Oct 30 22:30:00.950267 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:30:00.950240 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd1f7df_e348_4601_bfa9_18b26ef818cc.slice/crio-ace92b41907fdbf2c1599bb3dda3006c67041f4fcb33b81b3e0a1d8bb1ad3c62 WatchSource:0}: Error finding container ace92b41907fdbf2c1599bb3dda3006c67041f4fcb33b81b3e0a1d8bb1ad3c62: Status 404 returned error can't find the container with id ace92b41907fdbf2c1599bb3dda3006c67041f4fcb33b81b3e0a1d8bb1ad3c62 Oct 30 22:30:00.950581 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.950562 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq"] Oct 30 22:30:00.956460 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:00.956438 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp"] Oct 30 22:30:00.957061 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:30:00.957038 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80c8083_c781_47e2_ad75_9d2d7ab426b3.slice/crio-841f117de5376a779466d05348bccca92934714c784add07a7a1cb45805c5b5a WatchSource:0}: Error finding container 841f117de5376a779466d05348bccca92934714c784add07a7a1cb45805c5b5a: Status 404 returned error can't find the container with id 841f117de5376a779466d05348bccca92934714c784add07a7a1cb45805c5b5a Oct 30 22:30:01.001145 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:30:01.001120 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151b7632_44e1_4a13_a598_e454d11ef480.slice/crio-9f82dc21c41bc9912e1099a7a35a8e6305c21d20f56717c010c6cb8638e27582 WatchSource:0}: Error finding container 9f82dc21c41bc9912e1099a7a35a8e6305c21d20f56717c010c6cb8638e27582: Status 404 returned error can't find the container with id 9f82dc21c41bc9912e1099a7a35a8e6305c21d20f56717c010c6cb8638e27582 Oct 30 22:30:01.001801 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:01.001768 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm"] Oct 30 22:30:01.005624 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:01.005591 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5"] Oct 30 22:30:01.018282 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:01.018257 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" event={"ID":"b80c8083-c781-47e2-ad75-9d2d7ab426b3","Type":"ContainerStarted","Data":"841f117de5376a779466d05348bccca92934714c784add07a7a1cb45805c5b5a"} Oct 30 22:30:01.019773 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:01.019751 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" event={"ID":"ffd1f7df-e348-4601-bfa9-18b26ef818cc","Type":"ContainerStarted","Data":"ace92b41907fdbf2c1599bb3dda3006c67041f4fcb33b81b3e0a1d8bb1ad3c62"} Oct 30 22:30:01.021135 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:01.021112 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq" event={"ID":"5622d889-2cbb-4d12-8723-2211b3e79d35","Type":"ContainerStarted","Data":"94d3e45ccd772fa8e1f074002e863a6558171449357a3e7ba32aa04bdca096e1"} Oct 30 22:30:01.022464 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:01.022443 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" event={"ID":"151b7632-44e1-4a13-a598-e454d11ef480","Type":"ContainerStarted","Data":"9f82dc21c41bc9912e1099a7a35a8e6305c21d20f56717c010c6cb8638e27582"} Oct 30 22:30:01.100234 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:30:01.100207 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc1395f_6452_49d7_9452_272f05d50de1.slice/crio-2bb2d8a0239f36d2c9fa79ebe5137e451ee16b145e38adf9f79a0864cdab8594 WatchSource:0}: Error finding container 2bb2d8a0239f36d2c9fa79ebe5137e451ee16b145e38adf9f79a0864cdab8594: Status 404 returned error can't find the container with id 2bb2d8a0239f36d2c9fa79ebe5137e451ee16b145e38adf9f79a0864cdab8594 Oct 30 22:30:02.034379 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:02.034344 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" event={"ID":"151b7632-44e1-4a13-a598-e454d11ef480","Type":"ContainerStarted","Data":"59f16ddaa38f119a7023f1f2325640cfc24479cd8ffab30f4801a88216e3db87"} Oct 30 22:30:02.036344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:02.036311 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" event={"ID":"b80c8083-c781-47e2-ad75-9d2d7ab426b3","Type":"ContainerStarted","Data":"09c3f1bd0a49ef56c60ebba5eb81c8b9bd838738c91b8a2b309c250fffee6f4e"} Oct 30 22:30:02.038215 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:02.038191 4335 generic.go:356] "Generic (PLEG): container finished" podID="ffd1f7df-e348-4601-bfa9-18b26ef818cc" containerID="7cc35cbe04c84b2dd21fdb5e113f6b7c198eb3dee933a2b1f696e16d343dff98" exitCode=60 Oct 30 22:30:02.038327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:02.038276 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" event={"ID":"ffd1f7df-e348-4601-bfa9-18b26ef818cc","Type":"ContainerDied","Data":"7cc35cbe04c84b2dd21fdb5e113f6b7c198eb3dee933a2b1f696e16d343dff98"} Oct 30 22:30:02.038450 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:02.038433 4335 scope.go:122] "RemoveContainer" containerID="7cc35cbe04c84b2dd21fdb5e113f6b7c198eb3dee933a2b1f696e16d343dff98" Oct 30 22:30:02.040232 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:02.040212 4335 generic.go:356] "Generic (PLEG): container finished" podID="5622d889-2cbb-4d12-8723-2211b3e79d35" containerID="50c1a6acb6cf6b6d9a6ec110af7dd8697530fcfce61a7ce56107243d129cc15c" exitCode=60 Oct 30 22:30:02.040344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:02.040292 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq" event={"ID":"5622d889-2cbb-4d12-8723-2211b3e79d35","Type":"ContainerDied","Data":"50c1a6acb6cf6b6d9a6ec110af7dd8697530fcfce61a7ce56107243d129cc15c"} Oct 30 22:30:02.040497 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:02.040477 4335 scope.go:122] "RemoveContainer" containerID="50c1a6acb6cf6b6d9a6ec110af7dd8697530fcfce61a7ce56107243d129cc15c" Oct 30 22:30:02.042494 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:02.042272 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" event={"ID":"bdc1395f-6452-49d7-9452-272f05d50de1","Type":"ContainerStarted","Data":"693fac85f4aceffc712a99d5178bed4b40518015f572c2083e05987f4a8646bc"} Oct 30 22:30:02.042494 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:02.042309 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" event={"ID":"bdc1395f-6452-49d7-9452-272f05d50de1","Type":"ContainerStarted","Data":"2bb2d8a0239f36d2c9fa79ebe5137e451ee16b145e38adf9f79a0864cdab8594"} Oct 30 22:30:02.168590 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:02.168503 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" podStartSLOduration=2.168486005 podStartE2EDuration="2.168486005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.166710814 +0000 UTC m=+2735.267142554" watchObservedRunningTime="2026-10-30 22:30:02.168486005 +0000 UTC m=+2735.268917743" Oct 30 22:30:02.282929 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:02.282882 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" podStartSLOduration=2.282869659 podStartE2EDuration="2.282869659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.280269078 +0000 UTC m=+2735.380700814" watchObservedRunningTime="2026-10-30 22:30:02.282869659 +0000 UTC m=+2735.383301390" Oct 30 22:30:02.514261 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:02.514218 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" podStartSLOduration=2.514203872 podStartE2EDuration="2.514203872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.509748545 +0000 UTC m=+2735.610180277" watchObservedRunningTime="2026-10-30 22:30:02.514203872 +0000 UTC m=+2735.614635603" Oct 30 22:30:03.055502 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:03.055466 4335 generic.go:356] "Generic (PLEG): container finished" podID="b80c8083-c781-47e2-ad75-9d2d7ab426b3" containerID="09c3f1bd0a49ef56c60ebba5eb81c8b9bd838738c91b8a2b309c250fffee6f4e" exitCode=60 Oct 30 22:30:03.055986 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:03.055966 4335 scope.go:122] "RemoveContainer" containerID="09c3f1bd0a49ef56c60ebba5eb81c8b9bd838738c91b8a2b309c250fffee6f4e" Oct 30 22:30:03.061119 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:03.061087 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" event={"ID":"b80c8083-c781-47e2-ad75-9d2d7ab426b3","Type":"ContainerDied","Data":"09c3f1bd0a49ef56c60ebba5eb81c8b9bd838738c91b8a2b309c250fffee6f4e"} Oct 30 22:30:03.061231 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:03.061155 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" event={"ID":"ffd1f7df-e348-4601-bfa9-18b26ef818cc","Type":"ContainerStarted","Data":"f8b6cf5ea2710ebe0f9dd41b99f26914102e6f88693298860b86b86c8f87a123"} Oct 30 22:30:03.063428 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:03.063402 4335 generic.go:356] "Generic (PLEG): container finished" podID="5622d889-2cbb-4d12-8723-2211b3e79d35" containerID="0491e2a674e1cff662d41b7fb341ff7ae6fc77334d3bdbaad7633cbe120b6dcf" exitCode=60 Oct 30 22:30:03.063532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:03.063475 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq" event={"ID":"5622d889-2cbb-4d12-8723-2211b3e79d35","Type":"ContainerDied","Data":"0491e2a674e1cff662d41b7fb341ff7ae6fc77334d3bdbaad7633cbe120b6dcf"} Oct 30 22:30:03.063532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:03.063527 4335 scope.go:122] "RemoveContainer" containerID="50c1a6acb6cf6b6d9a6ec110af7dd8697530fcfce61a7ce56107243d129cc15c" Oct 30 22:30:03.063792 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:03.063777 4335 scope.go:122] "RemoveContainer" containerID="0491e2a674e1cff662d41b7fb341ff7ae6fc77334d3bdbaad7633cbe120b6dcf" Oct 30 22:30:03.064056 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:03.064030 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq_odh-ai-gateway-infra(5622d889-2cbb-4d12-8723-2211b3e79d35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq" podUID="5622d889-2cbb-4d12-8723-2211b3e79d35" Oct 30 22:30:03.065243 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:03.065222 4335 generic.go:356] "Generic (PLEG): container finished" podID="bdc1395f-6452-49d7-9452-272f05d50de1" containerID="693fac85f4aceffc712a99d5178bed4b40518015f572c2083e05987f4a8646bc" exitCode=60 Oct 30 22:30:03.065345 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:03.065302 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" event={"ID":"bdc1395f-6452-49d7-9452-272f05d50de1","Type":"ContainerDied","Data":"693fac85f4aceffc712a99d5178bed4b40518015f572c2083e05987f4a8646bc"} Oct 30 22:30:03.065500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:03.065486 4335 scope.go:122] "RemoveContainer" containerID="693fac85f4aceffc712a99d5178bed4b40518015f572c2083e05987f4a8646bc" Oct 30 22:30:03.067059 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:03.067039 4335 generic.go:356] "Generic (PLEG): container finished" podID="151b7632-44e1-4a13-a598-e454d11ef480" containerID="59f16ddaa38f119a7023f1f2325640cfc24479cd8ffab30f4801a88216e3db87" exitCode=60 Oct 30 22:30:03.067170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:03.067123 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" event={"ID":"151b7632-44e1-4a13-a598-e454d11ef480","Type":"ContainerDied","Data":"59f16ddaa38f119a7023f1f2325640cfc24479cd8ffab30f4801a88216e3db87"} Oct 30 22:30:03.067268 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:03.067254 4335 scope.go:122] "RemoveContainer" containerID="59f16ddaa38f119a7023f1f2325640cfc24479cd8ffab30f4801a88216e3db87" Oct 30 22:30:03.521314 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:03.521230 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" podStartSLOduration=3.521212336 podStartE2EDuration="3.521212336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.516726008 +0000 UTC m=+2736.617157741" watchObservedRunningTime="2026-10-30 22:30:03.521212336 +0000 UTC m=+2736.621644070" Oct 30 22:30:04.076984 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:04.076951 4335 generic.go:356] "Generic (PLEG): container finished" podID="ffd1f7df-e348-4601-bfa9-18b26ef818cc" containerID="f8b6cf5ea2710ebe0f9dd41b99f26914102e6f88693298860b86b86c8f87a123" exitCode=60 Oct 30 22:30:04.077453 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:04.077043 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" event={"ID":"ffd1f7df-e348-4601-bfa9-18b26ef818cc","Type":"ContainerDied","Data":"f8b6cf5ea2710ebe0f9dd41b99f26914102e6f88693298860b86b86c8f87a123"} Oct 30 22:30:04.077453 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:04.077106 4335 scope.go:122] "RemoveContainer" containerID="7cc35cbe04c84b2dd21fdb5e113f6b7c198eb3dee933a2b1f696e16d343dff98" Oct 30 22:30:04.077453 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:04.077211 4335 scope.go:122] "RemoveContainer" containerID="f8b6cf5ea2710ebe0f9dd41b99f26914102e6f88693298860b86b86c8f87a123" Oct 30 22:30:04.077453 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:04.077436 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq_odh-ai-gateway-infra(ffd1f7df-e348-4601-bfa9-18b26ef818cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" podUID="ffd1f7df-e348-4601-bfa9-18b26ef818cc" Oct 30 22:30:04.079184 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:04.079167 4335 scope.go:122] "RemoveContainer" containerID="0491e2a674e1cff662d41b7fb341ff7ae6fc77334d3bdbaad7633cbe120b6dcf" Oct 30 22:30:04.079394 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:04.079375 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq_odh-ai-gateway-infra(5622d889-2cbb-4d12-8723-2211b3e79d35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq" podUID="5622d889-2cbb-4d12-8723-2211b3e79d35" Oct 30 22:30:04.081277 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:04.081035 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" event={"ID":"bdc1395f-6452-49d7-9452-272f05d50de1","Type":"ContainerStarted","Data":"f039299a255f56f4195f74c8bca2b7962ae147e289c83bf955e7e6d34b8d1aaa"} Oct 30 22:30:04.083111 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:04.083089 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" event={"ID":"151b7632-44e1-4a13-a598-e454d11ef480","Type":"ContainerStarted","Data":"f2c188b277cad09dc7ccf820cb58d9bd9896af7b77071994adf8a56c8044caf0"} Oct 30 22:30:04.085227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:04.085205 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" event={"ID":"b80c8083-c781-47e2-ad75-9d2d7ab426b3","Type":"ContainerStarted","Data":"0d7b0ca53473e8b51b577fd57d23a538e6b6e2563c75dca6a202ab0eb8a99fd7"} Oct 30 22:30:05.094449 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:05.094418 4335 scope.go:122] "RemoveContainer" containerID="f8b6cf5ea2710ebe0f9dd41b99f26914102e6f88693298860b86b86c8f87a123" Oct 30 22:30:05.094949 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:05.094702 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq_odh-ai-gateway-infra(ffd1f7df-e348-4601-bfa9-18b26ef818cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" podUID="ffd1f7df-e348-4601-bfa9-18b26ef818cc" Oct 30 22:30:05.096296 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:05.096269 4335 generic.go:356] "Generic (PLEG): container finished" podID="bdc1395f-6452-49d7-9452-272f05d50de1" containerID="f039299a255f56f4195f74c8bca2b7962ae147e289c83bf955e7e6d34b8d1aaa" exitCode=60 Oct 30 22:30:05.096448 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:05.096354 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" event={"ID":"bdc1395f-6452-49d7-9452-272f05d50de1","Type":"ContainerDied","Data":"f039299a255f56f4195f74c8bca2b7962ae147e289c83bf955e7e6d34b8d1aaa"} Oct 30 22:30:05.096448 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:05.096415 4335 scope.go:122] "RemoveContainer" containerID="693fac85f4aceffc712a99d5178bed4b40518015f572c2083e05987f4a8646bc" Oct 30 22:30:05.096552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:05.096529 4335 scope.go:122] "RemoveContainer" containerID="f039299a255f56f4195f74c8bca2b7962ae147e289c83bf955e7e6d34b8d1aaa" Oct 30 22:30:05.096809 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:05.096772 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5_odh-ai-gateway-infra(bdc1395f-6452-49d7-9452-272f05d50de1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" podUID="bdc1395f-6452-49d7-9452-272f05d50de1" Oct 30 22:30:05.098683 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:05.098658 4335 generic.go:356] "Generic (PLEG): container finished" podID="151b7632-44e1-4a13-a598-e454d11ef480" containerID="f2c188b277cad09dc7ccf820cb58d9bd9896af7b77071994adf8a56c8044caf0" exitCode=60 Oct 30 22:30:05.098803 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:05.098725 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" event={"ID":"151b7632-44e1-4a13-a598-e454d11ef480","Type":"ContainerDied","Data":"f2c188b277cad09dc7ccf820cb58d9bd9896af7b77071994adf8a56c8044caf0"} Oct 30 22:30:05.098953 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:05.098932 4335 scope.go:122] "RemoveContainer" containerID="f2c188b277cad09dc7ccf820cb58d9bd9896af7b77071994adf8a56c8044caf0" Oct 30 22:30:05.099135 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:05.099114 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm_odh-ai-gateway-infra(151b7632-44e1-4a13-a598-e454d11ef480)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" podUID="151b7632-44e1-4a13-a598-e454d11ef480" Oct 30 22:30:05.100856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:05.100837 4335 generic.go:356] "Generic (PLEG): container finished" podID="b80c8083-c781-47e2-ad75-9d2d7ab426b3" containerID="0d7b0ca53473e8b51b577fd57d23a538e6b6e2563c75dca6a202ab0eb8a99fd7" exitCode=60 Oct 30 22:30:05.100976 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:05.100875 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" event={"ID":"b80c8083-c781-47e2-ad75-9d2d7ab426b3","Type":"ContainerDied","Data":"0d7b0ca53473e8b51b577fd57d23a538e6b6e2563c75dca6a202ab0eb8a99fd7"} Oct 30 22:30:05.101104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:05.101089 4335 scope.go:122] "RemoveContainer" containerID="0d7b0ca53473e8b51b577fd57d23a538e6b6e2563c75dca6a202ab0eb8a99fd7" Oct 30 22:30:05.101306 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:05.101289 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp_odh-ai-gateway-infra(b80c8083-c781-47e2-ad75-9d2d7ab426b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" podUID="b80c8083-c781-47e2-ad75-9d2d7ab426b3" Oct 30 22:30:05.110359 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:05.110332 4335 scope.go:122] "RemoveContainer" containerID="59f16ddaa38f119a7023f1f2325640cfc24479cd8ffab30f4801a88216e3db87" Oct 30 22:30:05.125757 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:05.125735 4335 scope.go:122] "RemoveContainer" containerID="09c3f1bd0a49ef56c60ebba5eb81c8b9bd838738c91b8a2b309c250fffee6f4e" Oct 30 22:30:06.112069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:06.112043 4335 scope.go:122] "RemoveContainer" containerID="0d7b0ca53473e8b51b577fd57d23a538e6b6e2563c75dca6a202ab0eb8a99fd7" Oct 30 22:30:06.112677 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:06.112316 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp_odh-ai-gateway-infra(b80c8083-c781-47e2-ad75-9d2d7ab426b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" podUID="b80c8083-c781-47e2-ad75-9d2d7ab426b3" Oct 30 22:30:06.113869 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:06.113850 4335 scope.go:122] "RemoveContainer" containerID="f039299a255f56f4195f74c8bca2b7962ae147e289c83bf955e7e6d34b8d1aaa" Oct 30 22:30:06.114127 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:06.114106 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5_odh-ai-gateway-infra(bdc1395f-6452-49d7-9452-272f05d50de1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" podUID="bdc1395f-6452-49d7-9452-272f05d50de1" Oct 30 22:30:06.115507 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:06.115483 4335 scope.go:122] "RemoveContainer" containerID="f2c188b277cad09dc7ccf820cb58d9bd9896af7b77071994adf8a56c8044caf0" Oct 30 22:30:06.115740 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:06.115721 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm_odh-ai-gateway-infra(151b7632-44e1-4a13-a598-e454d11ef480)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" podUID="151b7632-44e1-4a13-a598-e454d11ef480" Oct 30 22:30:16.050088 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:16.049995 4335 scope.go:122] "RemoveContainer" containerID="f8b6cf5ea2710ebe0f9dd41b99f26914102e6f88693298860b86b86c8f87a123" Oct 30 22:30:16.050549 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:16.050295 4335 scope.go:122] "RemoveContainer" containerID="f039299a255f56f4195f74c8bca2b7962ae147e289c83bf955e7e6d34b8d1aaa" Oct 30 22:30:16.050549 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:16.050460 4335 scope.go:122] "RemoveContainer" containerID="0491e2a674e1cff662d41b7fb341ff7ae6fc77334d3bdbaad7633cbe120b6dcf" Oct 30 22:30:17.055112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:17.055077 4335 scope.go:122] "RemoveContainer" containerID="f2c188b277cad09dc7ccf820cb58d9bd9896af7b77071994adf8a56c8044caf0" Oct 30 22:30:17.055733 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:17.055189 4335 scope.go:122] "RemoveContainer" containerID="0d7b0ca53473e8b51b577fd57d23a538e6b6e2563c75dca6a202ab0eb8a99fd7" Oct 30 22:30:17.200466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:17.200406 4335 generic.go:356] "Generic (PLEG): container finished" podID="ffd1f7df-e348-4601-bfa9-18b26ef818cc" containerID="ab28b62201386a38f4b14afddad5e24f911f42a8f6f513ec9bdaf881f6173aab" exitCode=60 Oct 30 22:30:17.200585 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:17.200479 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" event={"ID":"ffd1f7df-e348-4601-bfa9-18b26ef818cc","Type":"ContainerDied","Data":"ab28b62201386a38f4b14afddad5e24f911f42a8f6f513ec9bdaf881f6173aab"} Oct 30 22:30:17.200585 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:17.200532 4335 scope.go:122] "RemoveContainer" containerID="f8b6cf5ea2710ebe0f9dd41b99f26914102e6f88693298860b86b86c8f87a123" Oct 30 22:30:17.200922 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:17.200900 4335 scope.go:122] "RemoveContainer" containerID="ab28b62201386a38f4b14afddad5e24f911f42a8f6f513ec9bdaf881f6173aab" Oct 30 22:30:17.201173 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:17.201151 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq_odh-ai-gateway-infra(ffd1f7df-e348-4601-bfa9-18b26ef818cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" podUID="ffd1f7df-e348-4601-bfa9-18b26ef818cc" Oct 30 22:30:17.214474 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:17.214442 4335 generic.go:356] "Generic (PLEG): container finished" podID="5622d889-2cbb-4d12-8723-2211b3e79d35" containerID="cbe3aa155df54a36c28e55e5f479c89db9ad00c09cf002789d3cf877aa1dab6b" exitCode=60 Oct 30 22:30:17.214588 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:17.214546 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq" event={"ID":"5622d889-2cbb-4d12-8723-2211b3e79d35","Type":"ContainerDied","Data":"cbe3aa155df54a36c28e55e5f479c89db9ad00c09cf002789d3cf877aa1dab6b"} Oct 30 22:30:17.215056 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:17.215032 4335 scope.go:122] "RemoveContainer" containerID="cbe3aa155df54a36c28e55e5f479c89db9ad00c09cf002789d3cf877aa1dab6b" Oct 30 22:30:17.215425 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:17.215398 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq_odh-ai-gateway-infra(5622d889-2cbb-4d12-8723-2211b3e79d35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq" podUID="5622d889-2cbb-4d12-8723-2211b3e79d35" Oct 30 22:30:17.217845 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:17.217823 4335 generic.go:356] "Generic (PLEG): container finished" podID="bdc1395f-6452-49d7-9452-272f05d50de1" containerID="6412be53cb609baffbf0fe466622284fb7f591aa30d50f22017e2d81be73fe8e" exitCode=60 Oct 30 22:30:17.217953 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:17.217898 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" event={"ID":"bdc1395f-6452-49d7-9452-272f05d50de1","Type":"ContainerDied","Data":"6412be53cb609baffbf0fe466622284fb7f591aa30d50f22017e2d81be73fe8e"} Oct 30 22:30:17.218241 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:17.218224 4335 scope.go:122] "RemoveContainer" containerID="6412be53cb609baffbf0fe466622284fb7f591aa30d50f22017e2d81be73fe8e" Oct 30 22:30:17.218474 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:17.218453 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5_odh-ai-gateway-infra(bdc1395f-6452-49d7-9452-272f05d50de1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" podUID="bdc1395f-6452-49d7-9452-272f05d50de1" Oct 30 22:30:17.219836 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:17.219822 4335 scope.go:122] "RemoveContainer" containerID="0491e2a674e1cff662d41b7fb341ff7ae6fc77334d3bdbaad7633cbe120b6dcf" Oct 30 22:30:17.232934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:17.232912 4335 scope.go:122] "RemoveContainer" containerID="f039299a255f56f4195f74c8bca2b7962ae147e289c83bf955e7e6d34b8d1aaa" Oct 30 22:30:18.228048 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.228021 4335 generic.go:356] "Generic (PLEG): container finished" podID="151b7632-44e1-4a13-a598-e454d11ef480" containerID="e552db76b76380d613851dfe03295a60a661b6376ad792a72f1f2c9a703c5ba9" exitCode=60 Oct 30 22:30:18.228441 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.228098 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" event={"ID":"151b7632-44e1-4a13-a598-e454d11ef480","Type":"ContainerDied","Data":"e552db76b76380d613851dfe03295a60a661b6376ad792a72f1f2c9a703c5ba9"} Oct 30 22:30:18.228441 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.228165 4335 scope.go:122] "RemoveContainer" containerID="f2c188b277cad09dc7ccf820cb58d9bd9896af7b77071994adf8a56c8044caf0" Oct 30 22:30:18.228574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.228560 4335 scope.go:122] "RemoveContainer" containerID="e552db76b76380d613851dfe03295a60a661b6376ad792a72f1f2c9a703c5ba9" Oct 30 22:30:18.228895 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:18.228862 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm_odh-ai-gateway-infra(151b7632-44e1-4a13-a598-e454d11ef480)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" podUID="151b7632-44e1-4a13-a598-e454d11ef480" Oct 30 22:30:18.230330 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.230305 4335 generic.go:356] "Generic (PLEG): container finished" podID="b80c8083-c781-47e2-ad75-9d2d7ab426b3" containerID="861dfcd03ab1676966ee887999c471572b970c3269d49da8f21d2b1dbc43bac9" exitCode=60 Oct 30 22:30:18.230445 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.230373 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" event={"ID":"b80c8083-c781-47e2-ad75-9d2d7ab426b3","Type":"ContainerDied","Data":"861dfcd03ab1676966ee887999c471572b970c3269d49da8f21d2b1dbc43bac9"} Oct 30 22:30:18.230734 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.230716 4335 scope.go:122] "RemoveContainer" containerID="861dfcd03ab1676966ee887999c471572b970c3269d49da8f21d2b1dbc43bac9" Oct 30 22:30:18.230964 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:18.230942 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp_odh-ai-gateway-infra(b80c8083-c781-47e2-ad75-9d2d7ab426b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" podUID="b80c8083-c781-47e2-ad75-9d2d7ab426b3" Oct 30 22:30:18.242837 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.242814 4335 scope.go:122] "RemoveContainer" containerID="0d7b0ca53473e8b51b577fd57d23a538e6b6e2563c75dca6a202ab0eb8a99fd7" Oct 30 22:30:18.345195 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.345171 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq"] Oct 30 22:30:18.461474 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.461442 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq"] Oct 30 22:30:18.498149 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.498122 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" Oct 30 22:30:18.572866 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.572840 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5"] Oct 30 22:30:18.624491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.624465 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq" Oct 30 22:30:18.664581 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.664550 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd1f7df-e348-4601-bfa9-18b26ef818cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd1f7df-e348-4601-bfa9-18b26ef818cc-maas-api-service-ca\") pod \"ffd1f7df-e348-4601-bfa9-18b26ef818cc\" (UID: \"ffd1f7df-e348-4601-bfa9-18b26ef818cc\") " Oct 30 22:30:18.664755 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.664602 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd1f7df-e348-4601-bfa9-18b26ef818cc-kube-api-access-vph95\" (UniqueName: \"kubernetes.io/projected/ffd1f7df-e348-4601-bfa9-18b26ef818cc-kube-api-access-vph95\") pod \"ffd1f7df-e348-4601-bfa9-18b26ef818cc\" (UID: \"ffd1f7df-e348-4601-bfa9-18b26ef818cc\") " Oct 30 22:30:18.665010 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.664980 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd1f7df-e348-4601-bfa9-18b26ef818cc-maas-api-service-ca" pod "ffd1f7df-e348-4601-bfa9-18b26ef818cc" (UID: "ffd1f7df-e348-4601-bfa9-18b26ef818cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:18.665282 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.665257 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd1f7df-e348-4601-bfa9-18b26ef818cc-maas-api-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:18.666958 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.666931 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd1f7df-e348-4601-bfa9-18b26ef818cc-kube-api-access-vph95" pod "ffd1f7df-e348-4601-bfa9-18b26ef818cc" (UID: "ffd1f7df-e348-4601-bfa9-18b26ef818cc"). InnerVolumeSpecName "kube-api-access-vph95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:18.719166 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.719140 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" Oct 30 22:30:18.766255 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.766233 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5622d889-2cbb-4d12-8723-2211b3e79d35-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5622d889-2cbb-4d12-8723-2211b3e79d35-maas-api-service-ca\") pod \"5622d889-2cbb-4d12-8723-2211b3e79d35\" (UID: \"5622d889-2cbb-4d12-8723-2211b3e79d35\") " Oct 30 22:30:18.766378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.766285 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5622d889-2cbb-4d12-8723-2211b3e79d35-kube-api-access-x67bg\" (UniqueName: \"kubernetes.io/projected/5622d889-2cbb-4d12-8723-2211b3e79d35-kube-api-access-x67bg\") pod \"5622d889-2cbb-4d12-8723-2211b3e79d35\" (UID: \"5622d889-2cbb-4d12-8723-2211b3e79d35\") " Oct 30 22:30:18.766602 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.766589 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vph95\" (UniqueName: \"kubernetes.io/projected/ffd1f7df-e348-4601-bfa9-18b26ef818cc-kube-api-access-vph95\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:18.766690 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.766602 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5622d889-2cbb-4d12-8723-2211b3e79d35-maas-api-service-ca" pod "5622d889-2cbb-4d12-8723-2211b3e79d35" (UID: "5622d889-2cbb-4d12-8723-2211b3e79d35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:18.768359 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.768342 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5622d889-2cbb-4d12-8723-2211b3e79d35-kube-api-access-x67bg" pod "5622d889-2cbb-4d12-8723-2211b3e79d35" (UID: "5622d889-2cbb-4d12-8723-2211b3e79d35"). InnerVolumeSpecName "kube-api-access-x67bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:18.866991 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.866939 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc1395f-6452-49d7-9452-272f05d50de1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc1395f-6452-49d7-9452-272f05d50de1-maas-api-service-ca\") pod \"bdc1395f-6452-49d7-9452-272f05d50de1\" (UID: \"bdc1395f-6452-49d7-9452-272f05d50de1\") " Oct 30 22:30:18.867097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.867072 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc1395f-6452-49d7-9452-272f05d50de1-kube-api-access-xf7xp\" (UniqueName: \"kubernetes.io/projected/bdc1395f-6452-49d7-9452-272f05d50de1-kube-api-access-xf7xp\") pod \"bdc1395f-6452-49d7-9452-272f05d50de1\" (UID: \"bdc1395f-6452-49d7-9452-272f05d50de1\") " Oct 30 22:30:18.867252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.867229 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc1395f-6452-49d7-9452-272f05d50de1-maas-api-service-ca" pod "bdc1395f-6452-49d7-9452-272f05d50de1" (UID: "bdc1395f-6452-49d7-9452-272f05d50de1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:18.867483 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.867457 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc1395f-6452-49d7-9452-272f05d50de1-maas-api-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:18.867483 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.867481 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5622d889-2cbb-4d12-8723-2211b3e79d35-maas-api-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:18.867608 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.867495 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x67bg\" (UniqueName: \"kubernetes.io/projected/5622d889-2cbb-4d12-8723-2211b3e79d35-kube-api-access-x67bg\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:18.869215 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.869197 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc1395f-6452-49d7-9452-272f05d50de1-kube-api-access-xf7xp" pod "bdc1395f-6452-49d7-9452-272f05d50de1" (UID: "bdc1395f-6452-49d7-9452-272f05d50de1"). InnerVolumeSpecName "kube-api-access-xf7xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:18.968552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:18.968508 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf7xp\" (UniqueName: \"kubernetes.io/projected/bdc1395f-6452-49d7-9452-272f05d50de1-kube-api-access-xf7xp\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.245762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.245694 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq" Oct 30 22:30:19.245762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.245720 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq" event={"ID":"5622d889-2cbb-4d12-8723-2211b3e79d35","Type":"ContainerDied","Data":"94d3e45ccd772fa8e1f074002e863a6558171449357a3e7ba32aa04bdca096e1"} Oct 30 22:30:19.246253 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.245783 4335 scope.go:122] "RemoveContainer" containerID="cbe3aa155df54a36c28e55e5f479c89db9ad00c09cf002789d3cf877aa1dab6b" Oct 30 22:30:19.248151 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.248129 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" Oct 30 22:30:19.248266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.248214 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5" event={"ID":"bdc1395f-6452-49d7-9452-272f05d50de1","Type":"ContainerDied","Data":"2bb2d8a0239f36d2c9fa79ebe5137e451ee16b145e38adf9f79a0864cdab8594"} Oct 30 22:30:19.253334 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.253317 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" Oct 30 22:30:19.253424 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.253317 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq" event={"ID":"ffd1f7df-e348-4601-bfa9-18b26ef818cc","Type":"ContainerDied","Data":"ace92b41907fdbf2c1599bb3dda3006c67041f4fcb33b81b3e0a1d8bb1ad3c62"} Oct 30 22:30:19.258580 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.258564 4335 scope.go:122] "RemoveContainer" containerID="6412be53cb609baffbf0fe466622284fb7f591aa30d50f22017e2d81be73fe8e" Oct 30 22:30:19.268055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.268039 4335 scope.go:122] "RemoveContainer" containerID="ab28b62201386a38f4b14afddad5e24f911f42a8f6f513ec9bdaf881f6173aab" Oct 30 22:30:19.358560 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.358535 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm"] Oct 30 22:30:19.412876 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.412845 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq"] Oct 30 22:30:19.419286 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.418957 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c04440-29889990-xltbq"] Oct 30 22:30:19.472315 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.472285 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp"] Oct 30 22:30:19.513358 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.513298 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" Oct 30 22:30:19.525727 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.525703 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq"] Oct 30 22:30:19.529984 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.529961 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536975-29889990-wwnsq"] Oct 30 22:30:19.638521 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.638493 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5"] Oct 30 22:30:19.643354 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.643332 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32dd8d-29889990-cslv5"] Oct 30 22:30:19.650516 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.650497 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" Oct 30 22:30:19.676512 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.676490 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/151b7632-44e1-4a13-a598-e454d11ef480-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/151b7632-44e1-4a13-a598-e454d11ef480-maas-api-service-ca\") pod \"151b7632-44e1-4a13-a598-e454d11ef480\" (UID: \"151b7632-44e1-4a13-a598-e454d11ef480\") " Oct 30 22:30:19.676680 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.676643 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151b7632-44e1-4a13-a598-e454d11ef480-kube-api-access-pzhd4\" (UniqueName: \"kubernetes.io/projected/151b7632-44e1-4a13-a598-e454d11ef480-kube-api-access-pzhd4\") pod \"151b7632-44e1-4a13-a598-e454d11ef480\" (UID: \"151b7632-44e1-4a13-a598-e454d11ef480\") " Oct 30 22:30:19.676870 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.676851 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151b7632-44e1-4a13-a598-e454d11ef480-maas-api-service-ca" pod "151b7632-44e1-4a13-a598-e454d11ef480" (UID: "151b7632-44e1-4a13-a598-e454d11ef480"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:19.677004 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.676986 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/151b7632-44e1-4a13-a598-e454d11ef480-maas-api-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.678706 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.678687 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151b7632-44e1-4a13-a598-e454d11ef480-kube-api-access-pzhd4" pod "151b7632-44e1-4a13-a598-e454d11ef480" (UID: "151b7632-44e1-4a13-a598-e454d11ef480"). InnerVolumeSpecName "kube-api-access-pzhd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:19.777966 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.777891 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b80c8083-c781-47e2-ad75-9d2d7ab426b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b80c8083-c781-47e2-ad75-9d2d7ab426b3-maas-api-service-ca\") pod \"b80c8083-c781-47e2-ad75-9d2d7ab426b3\" (UID: \"b80c8083-c781-47e2-ad75-9d2d7ab426b3\") " Oct 30 22:30:19.778103 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.777992 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b80c8083-c781-47e2-ad75-9d2d7ab426b3-kube-api-access-8dnlj\" (UniqueName: \"kubernetes.io/projected/b80c8083-c781-47e2-ad75-9d2d7ab426b3-kube-api-access-8dnlj\") pod \"b80c8083-c781-47e2-ad75-9d2d7ab426b3\" (UID: \"b80c8083-c781-47e2-ad75-9d2d7ab426b3\") " Oct 30 22:30:19.778256 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.778236 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b80c8083-c781-47e2-ad75-9d2d7ab426b3-maas-api-service-ca" pod "b80c8083-c781-47e2-ad75-9d2d7ab426b3" (UID: "b80c8083-c781-47e2-ad75-9d2d7ab426b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:19.778490 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.778467 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b80c8083-c781-47e2-ad75-9d2d7ab426b3-maas-api-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.778597 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.778495 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzhd4\" (UniqueName: \"kubernetes.io/projected/151b7632-44e1-4a13-a598-e454d11ef480-kube-api-access-pzhd4\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.780234 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.780202 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80c8083-c781-47e2-ad75-9d2d7ab426b3-kube-api-access-8dnlj" pod "b80c8083-c781-47e2-ad75-9d2d7ab426b3" (UID: "b80c8083-c781-47e2-ad75-9d2d7ab426b3"). InnerVolumeSpecName "kube-api-access-8dnlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:19.879424 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:19.879400 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dnlj\" (UniqueName: \"kubernetes.io/projected/b80c8083-c781-47e2-ad75-9d2d7ab426b3-kube-api-access-8dnlj\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:20.262173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:20.262145 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" Oct 30 22:30:20.262173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:20.262163 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp" event={"ID":"b80c8083-c781-47e2-ad75-9d2d7ab426b3","Type":"ContainerDied","Data":"841f117de5376a779466d05348bccca92934714c784add07a7a1cb45805c5b5a"} Oct 30 22:30:20.262636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:20.262223 4335 scope.go:122] "RemoveContainer" containerID="861dfcd03ab1676966ee887999c471572b970c3269d49da8f21d2b1dbc43bac9" Oct 30 22:30:20.267657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:20.267593 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" event={"ID":"151b7632-44e1-4a13-a598-e454d11ef480","Type":"ContainerDied","Data":"9f82dc21c41bc9912e1099a7a35a8e6305c21d20f56717c010c6cb8638e27582"} Oct 30 22:30:20.267657 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:20.267644 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm" Oct 30 22:30:20.275789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:20.275769 4335 scope.go:122] "RemoveContainer" containerID="e552db76b76380d613851dfe03295a60a661b6376ad792a72f1f2c9a703c5ba9" Oct 30 22:30:20.431451 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:20.431425 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm"] Oct 30 22:30:20.437364 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:20.437342 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d1ca4b-29889990-f7zrm"] Oct 30 22:30:20.546880 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:20.546763 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp"] Oct 30 22:30:20.550922 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:20.550901 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-55beb8-29889990-5l4lp"] Oct 30 22:30:21.058842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:21.058811 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151b7632-44e1-4a13-a598-e454d11ef480" path="/var/lib/kubelet/pods/151b7632-44e1-4a13-a598-e454d11ef480/volumes" Oct 30 22:30:21.059212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:21.059198 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5622d889-2cbb-4d12-8723-2211b3e79d35" path="/var/lib/kubelet/pods/5622d889-2cbb-4d12-8723-2211b3e79d35/volumes" Oct 30 22:30:21.059556 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:21.059544 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b80c8083-c781-47e2-ad75-9d2d7ab426b3" path="/var/lib/kubelet/pods/b80c8083-c781-47e2-ad75-9d2d7ab426b3/volumes" Oct 30 22:30:21.060236 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:21.060221 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdc1395f-6452-49d7-9452-272f05d50de1" path="/var/lib/kubelet/pods/bdc1395f-6452-49d7-9452-272f05d50de1/volumes" Oct 30 22:30:21.060582 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:21.060570 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd1f7df-e348-4601-bfa9-18b26ef818cc" path="/var/lib/kubelet/pods/ffd1f7df-e348-4601-bfa9-18b26ef818cc/volumes" Oct 30 22:30:25.321832 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:25.321799 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:30:26.559834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:26.559801 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4"] Oct 30 22:30:26.560234 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:26.560025 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" podUID="c4116062-6db3-424f-b4cc-3442ab74a3b2" containerName="istio-proxy" containerID="cri-o://9b7a61f41dda4ce6157b4630ee8bde8f4e0ddc56161a7d3706c3d417f75f9752" gracePeriod=30 Oct 30 22:30:26.609562 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:26.609537 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j"] Oct 30 22:30:26.609745 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:26.609725 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" podUID="476ce668-42e8-41b9-aea8-9727298c9597" containerName="istio-proxy" containerID="cri-o://6d755986d7ec37b7cbc8d0576d130ec37f50df3f7b1b146c8ae61f9ee6e0d91d" gracePeriod=30 Oct 30 22:30:28.408391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.408358 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-blzr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:30:28.408807 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.408410 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" podUID="c4116062-6db3-424f-b4cc-3442ab74a3b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:30:28.652698 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.652658 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv"] Oct 30 22:30:28.653257 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653236 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5622d889-2cbb-4d12-8723-2211b3e79d35" containerName="cleanup" Oct 30 22:30:28.653378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653258 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5622d889-2cbb-4d12-8723-2211b3e79d35" containerName="cleanup" Oct 30 22:30:28.653378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653288 4335 cpu_manager.go:414] "containerMap: removing container" podUID="b80c8083-c781-47e2-ad75-9d2d7ab426b3" containerName="cleanup" Oct 30 22:30:28.653378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653296 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b80c8083-c781-47e2-ad75-9d2d7ab426b3" containerName="cleanup" Oct 30 22:30:28.653378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653304 4335 cpu_manager.go:414] "containerMap: removing container" podUID="b80c8083-c781-47e2-ad75-9d2d7ab426b3" containerName="cleanup" Oct 30 22:30:28.653378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653311 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b80c8083-c781-47e2-ad75-9d2d7ab426b3" containerName="cleanup" Oct 30 22:30:28.653378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653328 4335 cpu_manager.go:414] "containerMap: removing container" podUID="151b7632-44e1-4a13-a598-e454d11ef480" containerName="cleanup" Oct 30 22:30:28.653378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653334 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151b7632-44e1-4a13-a598-e454d11ef480" containerName="cleanup" Oct 30 22:30:28.653378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653347 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5622d889-2cbb-4d12-8723-2211b3e79d35" containerName="cleanup" Oct 30 22:30:28.653378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653353 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5622d889-2cbb-4d12-8723-2211b3e79d35" containerName="cleanup" Oct 30 22:30:28.653378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653365 4335 cpu_manager.go:414] "containerMap: removing container" podUID="b80c8083-c781-47e2-ad75-9d2d7ab426b3" containerName="cleanup" Oct 30 22:30:28.653378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653373 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b80c8083-c781-47e2-ad75-9d2d7ab426b3" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653393 4335 cpu_manager.go:414] "containerMap: removing container" podUID="bdc1395f-6452-49d7-9452-272f05d50de1" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653400 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc1395f-6452-49d7-9452-272f05d50de1" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653416 4335 cpu_manager.go:414] "containerMap: removing container" podUID="bdc1395f-6452-49d7-9452-272f05d50de1" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653423 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc1395f-6452-49d7-9452-272f05d50de1" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653442 4335 cpu_manager.go:414] "containerMap: removing container" podUID="ffd1f7df-e348-4601-bfa9-18b26ef818cc" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653449 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd1f7df-e348-4601-bfa9-18b26ef818cc" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653461 4335 cpu_manager.go:414] "containerMap: removing container" podUID="151b7632-44e1-4a13-a598-e454d11ef480" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653469 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151b7632-44e1-4a13-a598-e454d11ef480" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653482 4335 cpu_manager.go:414] "containerMap: removing container" podUID="ffd1f7df-e348-4601-bfa9-18b26ef818cc" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653490 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd1f7df-e348-4601-bfa9-18b26ef818cc" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653501 4335 cpu_manager.go:414] "containerMap: removing container" podUID="ffd1f7df-e348-4601-bfa9-18b26ef818cc" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653509 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd1f7df-e348-4601-bfa9-18b26ef818cc" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653667 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="b80c8083-c781-47e2-ad75-9d2d7ab426b3" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653683 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="b80c8083-c781-47e2-ad75-9d2d7ab426b3" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653693 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="151b7632-44e1-4a13-a598-e454d11ef480" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653703 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5622d889-2cbb-4d12-8723-2211b3e79d35" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653713 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5622d889-2cbb-4d12-8723-2211b3e79d35" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653723 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="b80c8083-c781-47e2-ad75-9d2d7ab426b3" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653733 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc1395f-6452-49d7-9452-272f05d50de1" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653743 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc1395f-6452-49d7-9452-272f05d50de1" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653753 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc1395f-6452-49d7-9452-272f05d50de1" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653762 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd1f7df-e348-4601-bfa9-18b26ef818cc" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653769 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="151b7632-44e1-4a13-a598-e454d11ef480" containerName="cleanup" Oct 30 22:30:28.654015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.653779 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd1f7df-e348-4601-bfa9-18b26ef818cc" containerName="cleanup" Oct 30 22:30:28.657547 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.657526 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.681856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.681816 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjh2f\" (UniqueName: \"kubernetes.io/projected/c5287819-aaee-4677-abb3-ee0f724b6d19-kube-api-access-bjh2f\") pod \"test-model-87b6b3-kserve-86957f4f79-kmbdv\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.681973 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.681859 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5287819-aaee-4677-abb3-ee0f724b6d19-tls-certs\") pod \"test-model-87b6b3-kserve-86957f4f79-kmbdv\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.682038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.681988 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-model-cache\") pod \"test-model-87b6b3-kserve-86957f4f79-kmbdv\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.682038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.682027 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-tmp-dir\") pod \"test-model-87b6b3-kserve-86957f4f79-kmbdv\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.682151 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.682049 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-home\") pod \"test-model-87b6b3-kserve-86957f4f79-kmbdv\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.682151 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.682107 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-dshm\") pod \"test-model-87b6b3-kserve-86957f4f79-kmbdv\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.759375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.759349 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-87b6b3\"/\"kube-root-ca.crt\"" Oct 30 22:30:28.759542 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.759360 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-87b6b3\"/\"openshift-service-ca.crt\"" Oct 30 22:30:28.759542 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.759418 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-87b6b3\"/\"default-dockercfg-qh474\"" Oct 30 22:30:28.759542 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.759449 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-87b6b3\"/\"test-model-87b6b3-kserve-self-signed-certs\"" Oct 30 22:30:28.763582 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.763563 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv"] Oct 30 22:30:28.783063 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.783034 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjh2f\" (UniqueName: \"kubernetes.io/projected/c5287819-aaee-4677-abb3-ee0f724b6d19-kube-api-access-bjh2f\") pod \"test-model-87b6b3-kserve-86957f4f79-kmbdv\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.783166 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.783067 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5287819-aaee-4677-abb3-ee0f724b6d19-tls-certs\") pod \"test-model-87b6b3-kserve-86957f4f79-kmbdv\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.783166 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.783112 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-model-cache\") pod \"test-model-87b6b3-kserve-86957f4f79-kmbdv\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.783166 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.783133 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-tmp-dir\") pod \"test-model-87b6b3-kserve-86957f4f79-kmbdv\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.783166 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.783148 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-home\") pod \"test-model-87b6b3-kserve-86957f4f79-kmbdv\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.783166 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.783165 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-dshm\") pod \"test-model-87b6b3-kserve-86957f4f79-kmbdv\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.783734 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.783539 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-model-cache\") pod \"test-model-87b6b3-kserve-86957f4f79-kmbdv\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.783734 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.783673 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-home\") pod \"test-model-87b6b3-kserve-86957f4f79-kmbdv\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.783734 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.783698 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-tmp-dir\") pod \"test-model-87b6b3-kserve-86957f4f79-kmbdv\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.787240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.787192 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-dshm\") pod \"test-model-87b6b3-kserve-86957f4f79-kmbdv\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.787820 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.787773 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5287819-aaee-4677-abb3-ee0f724b6d19-tls-certs\") pod \"test-model-87b6b3-kserve-86957f4f79-kmbdv\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.844356 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.844323 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjh2f\" (UniqueName: \"kubernetes.io/projected/c5287819-aaee-4677-abb3-ee0f724b6d19-kube-api-access-bjh2f\") pod \"test-model-87b6b3-kserve-86957f4f79-kmbdv\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:28.969599 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:28.969521 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:29.354287 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:29.354262 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv"] Oct 30 22:30:29.354964 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:30:29.354939 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5287819_aaee_4677_abb3_ee0f724b6d19.slice/crio-cddc4042bff80c8b0825a14a8804440fc4df862443d9e003a283db49fe71e820 WatchSource:0}: Error finding container cddc4042bff80c8b0825a14a8804440fc4df862443d9e003a283db49fe71e820: Status 404 returned error can't find the container with id cddc4042bff80c8b0825a14a8804440fc4df862443d9e003a283db49fe71e820 Oct 30 22:30:30.070227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:30.070199 4335 scope.go:122] "RemoveContainer" containerID="f9c9002c6e87715ba2b26115c4080ef2b2715381e1bdeef8be962d89afda461b" Oct 30 22:30:30.086811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:30.086761 4335 scope.go:122] "RemoveContainer" containerID="5d6c91c77538092fa9561857932c6ffbc64d3cd8e8e16f3360a7d11520711f4d" Oct 30 22:30:30.096902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:30.096871 4335 scope.go:122] "RemoveContainer" containerID="786ca8cf795716b58e62ae1fe50bab3e14e846f1233f2acec48b2fe18bf6d7e4" Oct 30 22:30:30.109240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:30.109221 4335 scope.go:122] "RemoveContainer" containerID="56344adb7fded7a6a3c8f246b2571cc2b69aa8fad4b0d74f7deffe05e947e3a3" Oct 30 22:30:30.119466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:30.119449 4335 scope.go:122] "RemoveContainer" containerID="e916455cc89f9a74ff46b2936652c2c771ed33180026f6e6fac3b03ded278e79" Oct 30 22:30:30.360469 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:30.360400 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" event={"ID":"c5287819-aaee-4677-abb3-ee0f724b6d19","Type":"ContainerStarted","Data":"7dd6af610e19290d5f1b1d54759aedd8fd8f5ff7d655dd3e3fa48a435e4a126f"} Oct 30 22:30:30.360469 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:30.360437 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" event={"ID":"c5287819-aaee-4677-abb3-ee0f724b6d19","Type":"ContainerStarted","Data":"cddc4042bff80c8b0825a14a8804440fc4df862443d9e003a283db49fe71e820"} Oct 30 22:30:30.478570 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:30.478520 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" podStartSLOduration=2.478507551 podStartE2EDuration="2.478507551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:30.476304134 +0000 UTC m=+2763.576735866" watchObservedRunningTime="2026-10-30 22:30:30.478507551 +0000 UTC m=+2763.578939326" Oct 30 22:30:30.717509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:30.717429 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-qt88j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:30:30.717509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:30.717483 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" podUID="476ce668-42e8-41b9-aea8-9727298c9597" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:30:30.970494 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:30.970412 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:31.629859 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:31.629829 4335 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4116062_6db3_424f_b4cc_3442ab74a3b2.slice/crio-9b7a61f41dda4ce6157b4630ee8bde8f4e0ddc56161a7d3706c3d417f75f9752.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:30:32.240629 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.240591 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:30:32.296075 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.296047 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:30:32.322087 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.322063 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4116062-6db3-424f-b4cc-3442ab74a3b2-kube-api-access-wgrc4\" (UniqueName: \"kubernetes.io/projected/c4116062-6db3-424f-b4cc-3442ab74a3b2-kube-api-access-wgrc4\") pod \"c4116062-6db3-424f-b4cc-3442ab74a3b2\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " Oct 30 22:30:32.322204 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.322148 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-data\") pod \"c4116062-6db3-424f-b4cc-3442ab74a3b2\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " Oct 30 22:30:32.322298 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.322207 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-credential-socket\") pod \"c4116062-6db3-424f-b4cc-3442ab74a3b2\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " Oct 30 22:30:32.322298 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.322243 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-envoy\") pod \"c4116062-6db3-424f-b4cc-3442ab74a3b2\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " Oct 30 22:30:32.322411 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.322296 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-workload-socket\") pod \"c4116062-6db3-424f-b4cc-3442ab74a3b2\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " Oct 30 22:30:32.322411 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.322333 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-token\" (UniqueName: \"kubernetes.io/projected/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-token\") pod \"c4116062-6db3-424f-b4cc-3442ab74a3b2\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " Oct 30 22:30:32.322411 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.322383 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4116062-6db3-424f-b4cc-3442ab74a3b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4116062-6db3-424f-b4cc-3442ab74a3b2-istiod-ca-cert\") pod \"c4116062-6db3-424f-b4cc-3442ab74a3b2\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " Oct 30 22:30:32.322572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.322393 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-data" pod "c4116062-6db3-424f-b4cc-3442ab74a3b2" (UID: "c4116062-6db3-424f-b4cc-3442ab74a3b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:32.322572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.322485 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-workload-certs\") pod \"c4116062-6db3-424f-b4cc-3442ab74a3b2\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " Oct 30 22:30:32.322572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.322499 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-credential-socket" pod "c4116062-6db3-424f-b4cc-3442ab74a3b2" (UID: "c4116062-6db3-424f-b4cc-3442ab74a3b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:32.322572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.322563 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-podinfo\") pod \"c4116062-6db3-424f-b4cc-3442ab74a3b2\" (UID: \"c4116062-6db3-424f-b4cc-3442ab74a3b2\") " Oct 30 22:30:32.322834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.322651 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-workload-socket" pod "c4116062-6db3-424f-b4cc-3442ab74a3b2" (UID: "c4116062-6db3-424f-b4cc-3442ab74a3b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:32.322834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.322769 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4116062-6db3-424f-b4cc-3442ab74a3b2-istiod-ca-cert" pod "c4116062-6db3-424f-b4cc-3442ab74a3b2" (UID: "c4116062-6db3-424f-b4cc-3442ab74a3b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:32.323150 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.323128 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:32.323279 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.323153 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:32.323279 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.323167 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:32.323279 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.323178 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4116062-6db3-424f-b4cc-3442ab74a3b2-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:32.323279 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.323247 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-workload-certs" pod "c4116062-6db3-424f-b4cc-3442ab74a3b2" (UID: "c4116062-6db3-424f-b4cc-3442ab74a3b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:32.324448 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.324423 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4116062-6db3-424f-b4cc-3442ab74a3b2-kube-api-access-wgrc4" pod "c4116062-6db3-424f-b4cc-3442ab74a3b2" (UID: "c4116062-6db3-424f-b4cc-3442ab74a3b2"). InnerVolumeSpecName "kube-api-access-wgrc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:32.324548 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.324526 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-envoy" pod "c4116062-6db3-424f-b4cc-3442ab74a3b2" (UID: "c4116062-6db3-424f-b4cc-3442ab74a3b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:32.324721 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.324697 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-podinfo" pod "c4116062-6db3-424f-b4cc-3442ab74a3b2" (UID: "c4116062-6db3-424f-b4cc-3442ab74a3b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:30:32.324943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.324928 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-token" pod "c4116062-6db3-424f-b4cc-3442ab74a3b2" (UID: "c4116062-6db3-424f-b4cc-3442ab74a3b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:32.377913 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.377863 4335 generic.go:356] "Generic (PLEG): container finished" podID="c4116062-6db3-424f-b4cc-3442ab74a3b2" containerID="9b7a61f41dda4ce6157b4630ee8bde8f4e0ddc56161a7d3706c3d417f75f9752" exitCode=0 Oct 30 22:30:32.378010 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.377915 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" Oct 30 22:30:32.378010 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.377961 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" event={"ID":"c4116062-6db3-424f-b4cc-3442ab74a3b2","Type":"ContainerDied","Data":"9b7a61f41dda4ce6157b4630ee8bde8f4e0ddc56161a7d3706c3d417f75f9752"} Oct 30 22:30:32.378104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.378027 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4" event={"ID":"c4116062-6db3-424f-b4cc-3442ab74a3b2","Type":"ContainerDied","Data":"c7a155080097f8799d7c6fa1bdec3eb44e938f6e28c5a208e5fe7b0b7004b389"} Oct 30 22:30:32.378104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.378064 4335 scope.go:122] "RemoveContainer" containerID="9b7a61f41dda4ce6157b4630ee8bde8f4e0ddc56161a7d3706c3d417f75f9752" Oct 30 22:30:32.379403 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.379381 4335 generic.go:356] "Generic (PLEG): container finished" podID="476ce668-42e8-41b9-aea8-9727298c9597" containerID="6d755986d7ec37b7cbc8d0576d130ec37f50df3f7b1b146c8ae61f9ee6e0d91d" exitCode=0 Oct 30 22:30:32.379510 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.379425 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" Oct 30 22:30:32.379510 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.379433 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" event={"ID":"476ce668-42e8-41b9-aea8-9727298c9597","Type":"ContainerDied","Data":"6d755986d7ec37b7cbc8d0576d130ec37f50df3f7b1b146c8ae61f9ee6e0d91d"} Oct 30 22:30:32.379510 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.379472 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j" event={"ID":"476ce668-42e8-41b9-aea8-9727298c9597","Type":"ContainerDied","Data":"c044ba54cb0f51d36f4e66778def9f627b31540ebe2538bf3167fe5c6dcdd9e0"} Oct 30 22:30:32.388806 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.388791 4335 scope.go:122] "RemoveContainer" containerID="9b7a61f41dda4ce6157b4630ee8bde8f4e0ddc56161a7d3706c3d417f75f9752" Oct 30 22:30:32.389078 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:32.389060 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7a61f41dda4ce6157b4630ee8bde8f4e0ddc56161a7d3706c3d417f75f9752\": container with ID starting with 9b7a61f41dda4ce6157b4630ee8bde8f4e0ddc56161a7d3706c3d417f75f9752 not found: ID does not exist" containerID="9b7a61f41dda4ce6157b4630ee8bde8f4e0ddc56161a7d3706c3d417f75f9752" Oct 30 22:30:32.389123 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.389087 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7a61f41dda4ce6157b4630ee8bde8f4e0ddc56161a7d3706c3d417f75f9752"} err="failed to get container status \"9b7a61f41dda4ce6157b4630ee8bde8f4e0ddc56161a7d3706c3d417f75f9752\": rpc error: code = NotFound desc = could not find container \"9b7a61f41dda4ce6157b4630ee8bde8f4e0ddc56161a7d3706c3d417f75f9752\": container with ID starting with 9b7a61f41dda4ce6157b4630ee8bde8f4e0ddc56161a7d3706c3d417f75f9752 not found: ID does not exist" Oct 30 22:30:32.389123 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.389103 4335 scope.go:122] "RemoveContainer" containerID="6d755986d7ec37b7cbc8d0576d130ec37f50df3f7b1b146c8ae61f9ee6e0d91d" Oct 30 22:30:32.398962 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.398946 4335 scope.go:122] "RemoveContainer" containerID="6d755986d7ec37b7cbc8d0576d130ec37f50df3f7b1b146c8ae61f9ee6e0d91d" Oct 30 22:30:32.399209 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:32.399191 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d755986d7ec37b7cbc8d0576d130ec37f50df3f7b1b146c8ae61f9ee6e0d91d\": container with ID starting with 6d755986d7ec37b7cbc8d0576d130ec37f50df3f7b1b146c8ae61f9ee6e0d91d not found: ID does not exist" containerID="6d755986d7ec37b7cbc8d0576d130ec37f50df3f7b1b146c8ae61f9ee6e0d91d" Oct 30 22:30:32.399269 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.399217 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d755986d7ec37b7cbc8d0576d130ec37f50df3f7b1b146c8ae61f9ee6e0d91d"} err="failed to get container status \"6d755986d7ec37b7cbc8d0576d130ec37f50df3f7b1b146c8ae61f9ee6e0d91d\": rpc error: code = NotFound desc = could not find container \"6d755986d7ec37b7cbc8d0576d130ec37f50df3f7b1b146c8ae61f9ee6e0d91d\": container with ID starting with 6d755986d7ec37b7cbc8d0576d130ec37f50df3f7b1b146c8ae61f9ee6e0d91d not found: ID does not exist" Oct 30 22:30:32.423715 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.423688 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/476ce668-42e8-41b9-aea8-9727298c9597-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476ce668-42e8-41b9-aea8-9727298c9597-istio-podinfo\") pod \"476ce668-42e8-41b9-aea8-9727298c9597\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " Oct 30 22:30:32.423814 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.423747 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-workload-socket\") pod \"476ce668-42e8-41b9-aea8-9727298c9597\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " Oct 30 22:30:32.423814 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.423775 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/476ce668-42e8-41b9-aea8-9727298c9597-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476ce668-42e8-41b9-aea8-9727298c9597-istiod-ca-cert\") pod \"476ce668-42e8-41b9-aea8-9727298c9597\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " Oct 30 22:30:32.423939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.423824 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-workload-certs\") pod \"476ce668-42e8-41b9-aea8-9727298c9597\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " Oct 30 22:30:32.423939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.423904 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476ce668-42e8-41b9-aea8-9727298c9597-istio-token\" (UniqueName: \"kubernetes.io/projected/476ce668-42e8-41b9-aea8-9727298c9597-istio-token\") pod \"476ce668-42e8-41b9-aea8-9727298c9597\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " Oct 30 22:30:32.424048 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.423959 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-credential-socket\") pod \"476ce668-42e8-41b9-aea8-9727298c9597\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " Oct 30 22:30:32.424048 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.424004 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476ce668-42e8-41b9-aea8-9727298c9597-kube-api-access-28nbr\" (UniqueName: \"kubernetes.io/projected/476ce668-42e8-41b9-aea8-9727298c9597-kube-api-access-28nbr\") pod \"476ce668-42e8-41b9-aea8-9727298c9597\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " Oct 30 22:30:32.424225 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.424202 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-workload-certs" pod "476ce668-42e8-41b9-aea8-9727298c9597" (UID: "476ce668-42e8-41b9-aea8-9727298c9597"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:32.424493 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.424470 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476ce668-42e8-41b9-aea8-9727298c9597-istiod-ca-cert" pod "476ce668-42e8-41b9-aea8-9727298c9597" (UID: "476ce668-42e8-41b9-aea8-9727298c9597"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:32.425572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.424563 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-credential-socket" pod "476ce668-42e8-41b9-aea8-9727298c9597" (UID: "476ce668-42e8-41b9-aea8-9727298c9597"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:32.425572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.424635 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-istio-envoy\") pod \"476ce668-42e8-41b9-aea8-9727298c9597\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " Oct 30 22:30:32.425572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.424684 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-workload-socket" pod "476ce668-42e8-41b9-aea8-9727298c9597" (UID: "476ce668-42e8-41b9-aea8-9727298c9597"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:32.425572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.424711 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-istio-data\") pod \"476ce668-42e8-41b9-aea8-9727298c9597\" (UID: \"476ce668-42e8-41b9-aea8-9727298c9597\") " Oct 30 22:30:32.425572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.425161 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgrc4\" (UniqueName: \"kubernetes.io/projected/c4116062-6db3-424f-b4cc-3442ab74a3b2-kube-api-access-wgrc4\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:32.425572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.425180 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:32.425572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.425194 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:32.425572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.425209 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:32.425572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.425222 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4116062-6db3-424f-b4cc-3442ab74a3b2-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:32.425572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.425237 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:32.425572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.425251 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476ce668-42e8-41b9-aea8-9727298c9597-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:32.425572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.425264 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:32.425572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.425277 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4116062-6db3-424f-b4cc-3442ab74a3b2-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:32.425572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.425235 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-istio-data" pod "476ce668-42e8-41b9-aea8-9727298c9597" (UID: "476ce668-42e8-41b9-aea8-9727298c9597"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:32.426666 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.426630 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/476ce668-42e8-41b9-aea8-9727298c9597-istio-podinfo" pod "476ce668-42e8-41b9-aea8-9727298c9597" (UID: "476ce668-42e8-41b9-aea8-9727298c9597"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:30:32.426991 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.426960 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476ce668-42e8-41b9-aea8-9727298c9597-istio-token" pod "476ce668-42e8-41b9-aea8-9727298c9597" (UID: "476ce668-42e8-41b9-aea8-9727298c9597"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:32.427536 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.427508 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-istio-envoy" pod "476ce668-42e8-41b9-aea8-9727298c9597" (UID: "476ce668-42e8-41b9-aea8-9727298c9597"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:32.427751 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.427733 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476ce668-42e8-41b9-aea8-9727298c9597-kube-api-access-28nbr" pod "476ce668-42e8-41b9-aea8-9727298c9597" (UID: "476ce668-42e8-41b9-aea8-9727298c9597"). InnerVolumeSpecName "kube-api-access-28nbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:32.522942 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.522917 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4"] Oct 30 22:30:32.526634 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.526594 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476ce668-42e8-41b9-aea8-9727298c9597-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:32.526738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.526648 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476ce668-42e8-41b9-aea8-9727298c9597-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:32.526738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.526659 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28nbr\" (UniqueName: \"kubernetes.io/projected/476ce668-42e8-41b9-aea8-9727298c9597-kube-api-access-28nbr\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:32.526738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.526670 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:32.526738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.526679 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476ce668-42e8-41b9-aea8-9727298c9597-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:32.527202 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.527183 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-blzr4"] Oct 30 22:30:32.802114 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.802083 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j"] Oct 30 22:30:32.807591 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:32.807565 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qt88j"] Oct 30 22:30:33.057523 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:33.057496 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="476ce668-42e8-41b9-aea8-9727298c9597" path="/var/lib/kubelet/pods/476ce668-42e8-41b9-aea8-9727298c9597/volumes" Oct 30 22:30:33.058036 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:33.058019 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4116062-6db3-424f-b4cc-3442ab74a3b2" path="/var/lib/kubelet/pods/c4116062-6db3-424f-b4cc-3442ab74a3b2/volumes" Oct 30 22:30:37.089566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:37.089534 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:30:38.970526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:38.970479 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:38.983670 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:38.983648 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:39.456890 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:39.456858 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:30:55.322698 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:30:55.322658 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:30:59.057939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:59.057898 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb"] Oct 30 22:30:59.100863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:59.100822 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9"] Oct 30 22:30:59.104266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:59.104240 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14dca27-hmbtb"] Oct 30 22:30:59.107037 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:30:59.107014 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81565b10-gwgm9"] Oct 30 22:31:01.057936 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:31:01.057900 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97d854e6-f96e-4c8f-906a-9f2a46eabc35" path="/var/lib/kubelet/pods/97d854e6-f96e-4c8f-906a-9f2a46eabc35/volumes" Oct 30 22:31:01.058455 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:31:01.058439 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1655cc1-72e4-4f89-a69f-b351d3228a1c" path="/var/lib/kubelet/pods/b1655cc1-72e4-4f89-a69f-b351d3228a1c/volumes" Oct 30 22:31:24.271097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:31:24.271063 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:31:25.320862 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:31:25.320821 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:31:30.195838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:31:30.195799 4335 scope.go:122] "RemoveContainer" containerID="e6696dcc4a73dd462008a75969a10dbcd6b8fbf8d4d179cf191a7cd37dff035a" Oct 30 22:31:30.213414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:31:30.213393 4335 scope.go:122] "RemoveContainer" containerID="6f640a772d55fad8f12ca784c6d78ac50fc7422c5dbeb0f5e2f33f3bd7d993fd" Oct 30 22:31:30.225127 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:31:30.225101 4335 scope.go:122] "RemoveContainer" containerID="2a3decaa595b0be204ab1359271dfc87fbd2862d6351edf3362bc1b8a44119e5" Oct 30 22:31:30.236883 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:31:30.236865 4335 scope.go:122] "RemoveContainer" containerID="42ee60300ddfd1954b7068feb443bb30cf5cc25220dea0111910611dbadd994d" Oct 30 22:31:30.245694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:31:30.245676 4335 scope.go:122] "RemoveContainer" containerID="e5c4bd93cc9311e2cfa3c2dd6509dda274c88e51831ff8e356a621d466a14fe2" Oct 30 22:31:39.049499 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:31:39.049424 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2"] Oct 30 22:31:39.099336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:31:39.099307 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb"] Oct 30 22:31:39.102666 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:31:39.102648 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d058665-w2st2"] Oct 30 22:31:39.105688 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:31:39.105667 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b61b44bd-g5hbb"] Oct 30 22:31:41.057865 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:31:41.057833 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f9df26f-a88b-431d-b6dc-d608694b1b55" path="/var/lib/kubelet/pods/2f9df26f-a88b-431d-b6dc-d608694b1b55/volumes" Oct 30 22:31:41.058331 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:31:41.058315 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97439aa6-0ced-4c9b-bd2e-73039917e317" path="/var/lib/kubelet/pods/97439aa6-0ced-4c9b-bd2e-73039917e317/volumes" Oct 30 22:31:55.320671 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:31:55.320606 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:32:01.238240 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:32:01.238196 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:32:25.321421 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:32:25.321380 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:32:30.300465 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:32:30.300434 4335 scope.go:122] "RemoveContainer" containerID="5996082feac6897b6ed100acdb21f72401dd3a33d774196c2a0c56272fbfb5ce" Oct 30 22:32:30.312028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:32:30.311999 4335 scope.go:122] "RemoveContainer" containerID="e5a998d8202c0969d0a7ec9a9d31b8199d7b221b65089a29a932adb77e9e1785" Oct 30 22:32:30.326036 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:32:30.326018 4335 scope.go:122] "RemoveContainer" containerID="882ece59432e9b9a14a7e6e0720296744ca8b67ce3ac7671d1ff8dfb292303a5" Oct 30 22:32:30.335945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:32:30.335908 4335 scope.go:122] "RemoveContainer" containerID="6aa4874772db37fa397a475fb3343660f562623fed8de034b96495379a6bf7d2" Oct 30 22:32:30.346491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:32:30.346469 4335 scope.go:122] "RemoveContainer" containerID="923e9e70cabd94e43191170017afa399fa2b14cefa3df3689386f24466c25558" Oct 30 22:32:55.320479 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:32:55.320447 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:33:08.646438 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:08.646386 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:33:11.065361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:11.065325 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f"] Oct 30 22:33:11.114689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:11.114660 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a87cf99-c6c6f"] Oct 30 22:33:13.058098 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:13.058059 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd40b3cc-a54a-4d37-957e-40715570dbbf" path="/var/lib/kubelet/pods/bd40b3cc-a54a-4d37-957e-40715570dbbf/volumes" Oct 30 22:33:16.073281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:16.073247 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm"] Oct 30 22:33:17.058485 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:17.058457 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l"] Oct 30 22:33:17.100739 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:17.100710 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d5fc2b0-5727l"] Oct 30 22:33:19.057727 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:19.057685 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73b4c13f-c140-4d54-a269-317da161c851" path="/var/lib/kubelet/pods/73b4c13f-c140-4d54-a269-317da161c851/volumes" Oct 30 22:33:25.322175 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:33:25.322137 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:33:30.392098 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:30.392058 4335 scope.go:122] "RemoveContainer" containerID="3f21a62b396225d2e85ff508ba16d64d2cb433708e904565d233c4f28292fe3b" Oct 30 22:33:30.403590 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:30.403566 4335 scope.go:122] "RemoveContainer" containerID="e7b576c666b5920a28a40d688053f4abd2d847cde69dc86985ec23ca585a1087" Oct 30 22:33:31.093647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.093568 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" podUID="b9e3ce96-8e7f-410f-b717-b6651336c373" containerName="main" containerID="cri-o://ccacbf1b2dcdf9f7e68b5cfd9332eb288ea126d3d731b04bc44f80d2e5f777e6" gracePeriod=45 Oct 30 22:33:31.369774 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.369750 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:33:31.475819 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.475793 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-tmp-dir\") pod \"b9e3ce96-8e7f-410f-b717-b6651336c373\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " Oct 30 22:33:31.476126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.475826 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-model-cache\") pod \"b9e3ce96-8e7f-410f-b717-b6651336c373\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " Oct 30 22:33:31.476126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.475971 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e3ce96-8e7f-410f-b717-b6651336c373-kube-api-access-cjwj7\" (UniqueName: \"kubernetes.io/projected/b9e3ce96-8e7f-410f-b717-b6651336c373-kube-api-access-cjwj7\") pod \"b9e3ce96-8e7f-410f-b717-b6651336c373\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " Oct 30 22:33:31.476126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.476013 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-dshm\") pod \"b9e3ce96-8e7f-410f-b717-b6651336c373\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " Oct 30 22:33:31.476126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.476058 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-tmp-dir" pod "b9e3ce96-8e7f-410f-b717-b6651336c373" (UID: "b9e3ce96-8e7f-410f-b717-b6651336c373"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:31.476126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.476085 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-home\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-home\") pod \"b9e3ce96-8e7f-410f-b717-b6651336c373\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " Oct 30 22:33:31.476429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.476131 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9e3ce96-8e7f-410f-b717-b6651336c373-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e3ce96-8e7f-410f-b717-b6651336c373-tls-certs\") pod \"b9e3ce96-8e7f-410f-b717-b6651336c373\" (UID: \"b9e3ce96-8e7f-410f-b717-b6651336c373\") " Oct 30 22:33:31.476429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.476145 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-model-cache" pod "b9e3ce96-8e7f-410f-b717-b6651336c373" (UID: "b9e3ce96-8e7f-410f-b717-b6651336c373"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:31.476429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.476382 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-home" pod "b9e3ce96-8e7f-410f-b717-b6651336c373" (UID: "b9e3ce96-8e7f-410f-b717-b6651336c373"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:31.476595 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.476552 4335 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-home\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:31.476595 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.476571 4335 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-tmp-dir\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:31.476595 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.476585 4335 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-model-cache\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:31.478305 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.478283 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9e3ce96-8e7f-410f-b717-b6651336c373-tls-certs" pod "b9e3ce96-8e7f-410f-b717-b6651336c373" (UID: "b9e3ce96-8e7f-410f-b717-b6651336c373"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:31.478406 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.478321 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-dshm" pod "b9e3ce96-8e7f-410f-b717-b6651336c373" (UID: "b9e3ce96-8e7f-410f-b717-b6651336c373"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:31.478406 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.478360 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e3ce96-8e7f-410f-b717-b6651336c373-kube-api-access-cjwj7" pod "b9e3ce96-8e7f-410f-b717-b6651336c373" (UID: "b9e3ce96-8e7f-410f-b717-b6651336c373"). InnerVolumeSpecName "kube-api-access-cjwj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:31.577862 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.577832 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjwj7\" (UniqueName: \"kubernetes.io/projected/b9e3ce96-8e7f-410f-b717-b6651336c373-kube-api-access-cjwj7\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:31.577862 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.577855 4335 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e3ce96-8e7f-410f-b717-b6651336c373-dshm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:31.577862 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.577865 4335 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e3ce96-8e7f-410f-b717-b6651336c373-tls-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:31.731477 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.731406 4335 generic.go:356] "Generic (PLEG): container finished" podID="b9e3ce96-8e7f-410f-b717-b6651336c373" containerID="ccacbf1b2dcdf9f7e68b5cfd9332eb288ea126d3d731b04bc44f80d2e5f777e6" exitCode=0 Oct 30 22:33:31.731477 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.731459 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" Oct 30 22:33:31.731738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.731493 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" event={"ID":"b9e3ce96-8e7f-410f-b717-b6651336c373","Type":"ContainerDied","Data":"ccacbf1b2dcdf9f7e68b5cfd9332eb288ea126d3d731b04bc44f80d2e5f777e6"} Oct 30 22:33:31.731738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.731545 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm" event={"ID":"b9e3ce96-8e7f-410f-b717-b6651336c373","Type":"ContainerDied","Data":"97cec7493da509a54ceb0381d8d64580b2b7bfd8d9e4c4012690a7d2ee431f75"} Oct 30 22:33:31.731738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.731571 4335 scope.go:122] "RemoveContainer" containerID="ccacbf1b2dcdf9f7e68b5cfd9332eb288ea126d3d731b04bc44f80d2e5f777e6" Oct 30 22:33:31.743046 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.743023 4335 scope.go:122] "RemoveContainer" containerID="ccacbf1b2dcdf9f7e68b5cfd9332eb288ea126d3d731b04bc44f80d2e5f777e6" Oct 30 22:33:31.743304 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:33:31.743286 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccacbf1b2dcdf9f7e68b5cfd9332eb288ea126d3d731b04bc44f80d2e5f777e6\": container with ID starting with ccacbf1b2dcdf9f7e68b5cfd9332eb288ea126d3d731b04bc44f80d2e5f777e6 not found: ID does not exist" containerID="ccacbf1b2dcdf9f7e68b5cfd9332eb288ea126d3d731b04bc44f80d2e5f777e6" Oct 30 22:33:31.743357 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.743313 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccacbf1b2dcdf9f7e68b5cfd9332eb288ea126d3d731b04bc44f80d2e5f777e6"} err="failed to get container status \"ccacbf1b2dcdf9f7e68b5cfd9332eb288ea126d3d731b04bc44f80d2e5f777e6\": rpc error: code = NotFound desc = could not find container \"ccacbf1b2dcdf9f7e68b5cfd9332eb288ea126d3d731b04bc44f80d2e5f777e6\": container with ID starting with ccacbf1b2dcdf9f7e68b5cfd9332eb288ea126d3d731b04bc44f80d2e5f777e6 not found: ID does not exist" Oct 30 22:33:31.897367 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.897339 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm"] Oct 30 22:33:31.900706 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:31.900687 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-026c11/test-model-026c11-kserve-5c6f4d5989-c88hm"] Oct 30 22:33:33.058449 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:33.058409 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9e3ce96-8e7f-410f-b717-b6651336c373" path="/var/lib/kubelet/pods/b9e3ce96-8e7f-410f-b717-b6651336c373/volumes" Oct 30 22:33:36.961910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:36.961879 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:33:55.321978 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:33:55.321935 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:33:57.217682 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:57.217647 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv"] Oct 30 22:33:58.347506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.347475 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8"] Oct 30 22:33:58.348020 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.348006 4335 cpu_manager.go:414] "containerMap: removing container" podUID="476ce668-42e8-41b9-aea8-9727298c9597" containerName="istio-proxy" Oct 30 22:33:58.348067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.348021 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476ce668-42e8-41b9-aea8-9727298c9597" containerName="istio-proxy" Oct 30 22:33:58.348067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.348043 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5622d889-2cbb-4d12-8723-2211b3e79d35" containerName="cleanup" Oct 30 22:33:58.348067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.348049 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5622d889-2cbb-4d12-8723-2211b3e79d35" containerName="cleanup" Oct 30 22:33:58.348067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.348060 4335 cpu_manager.go:414] "containerMap: removing container" podUID="b9e3ce96-8e7f-410f-b717-b6651336c373" containerName="main" Oct 30 22:33:58.348067 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.348066 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9e3ce96-8e7f-410f-b717-b6651336c373" containerName="main" Oct 30 22:33:58.348229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.348077 4335 cpu_manager.go:414] "containerMap: removing container" podUID="bdc1395f-6452-49d7-9452-272f05d50de1" containerName="cleanup" Oct 30 22:33:58.348229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.348082 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc1395f-6452-49d7-9452-272f05d50de1" containerName="cleanup" Oct 30 22:33:58.348229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.348090 4335 cpu_manager.go:414] "containerMap: removing container" podUID="c4116062-6db3-424f-b4cc-3442ab74a3b2" containerName="istio-proxy" Oct 30 22:33:58.348229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.348095 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4116062-6db3-424f-b4cc-3442ab74a3b2" containerName="istio-proxy" Oct 30 22:33:58.348229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.348109 4335 cpu_manager.go:414] "containerMap: removing container" podUID="151b7632-44e1-4a13-a598-e454d11ef480" containerName="cleanup" Oct 30 22:33:58.348229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.348114 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151b7632-44e1-4a13-a598-e454d11ef480" containerName="cleanup" Oct 30 22:33:58.348229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.348190 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="151b7632-44e1-4a13-a598-e454d11ef480" containerName="cleanup" Oct 30 22:33:58.348229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.348201 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5622d889-2cbb-4d12-8723-2211b3e79d35" containerName="cleanup" Oct 30 22:33:58.348229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.348209 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9e3ce96-8e7f-410f-b717-b6651336c373" containerName="main" Oct 30 22:33:58.348229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.348216 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4116062-6db3-424f-b4cc-3442ab74a3b2" containerName="istio-proxy" Oct 30 22:33:58.348229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.348223 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd1f7df-e348-4601-bfa9-18b26ef818cc" containerName="cleanup" Oct 30 22:33:58.348229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.348232 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="476ce668-42e8-41b9-aea8-9727298c9597" containerName="istio-proxy" Oct 30 22:33:58.351711 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.351691 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8" Oct 30 22:33:58.362774 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.362751 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8"] Oct 30 22:33:58.403935 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.403904 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:33:58.403935 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.403934 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:33:58.412915 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.412887 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:33:58.412915 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.412896 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:33:58.447055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.447028 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps65q\" (UniqueName: \"kubernetes.io/projected/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-kube-api-access-ps65q\") pod \"maas-api-revoke-sub-ee3bd233-c2xc8\" (UID: \"20af7c9c-26cb-4ae1-b572-8a1ed8525b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8" Oct 30 22:33:58.447183 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.447079 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee3bd233-c2xc8\" (UID: \"20af7c9c-26cb-4ae1-b572-8a1ed8525b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8" Oct 30 22:33:58.447236 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.447218 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee3bd233-c2xc8\" (UID: \"20af7c9c-26cb-4ae1-b572-8a1ed8525b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8" Oct 30 22:33:58.464156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.464128 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:33:58.547900 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.547873 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps65q\" (UniqueName: \"kubernetes.io/projected/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-kube-api-access-ps65q\") pod \"maas-api-revoke-sub-ee3bd233-c2xc8\" (UID: \"20af7c9c-26cb-4ae1-b572-8a1ed8525b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8" Oct 30 22:33:58.548041 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.547907 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee3bd233-c2xc8\" (UID: \"20af7c9c-26cb-4ae1-b572-8a1ed8525b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8" Oct 30 22:33:58.548041 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.547964 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee3bd233-c2xc8\" (UID: \"20af7c9c-26cb-4ae1-b572-8a1ed8525b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8" Oct 30 22:33:58.548809 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.548783 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee3bd233-c2xc8\" (UID: \"20af7c9c-26cb-4ae1-b572-8a1ed8525b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8" Oct 30 22:33:58.550550 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.550529 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee3bd233-c2xc8\" (UID: \"20af7c9c-26cb-4ae1-b572-8a1ed8525b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8" Oct 30 22:33:58.609375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.609283 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps65q\" (UniqueName: \"kubernetes.io/projected/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-kube-api-access-ps65q\") pod \"maas-api-revoke-sub-ee3bd233-c2xc8\" (UID: \"20af7c9c-26cb-4ae1-b572-8a1ed8525b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8" Oct 30 22:33:58.672402 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:58.672371 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8" Oct 30 22:33:59.056306 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:33:59.056267 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20af7c9c_26cb_4ae1_b572_8a1ed8525b28.slice/crio-027056d92ce45d4005f0e87defc317608d19ccdee532398a4de0e870d806d668 WatchSource:0}: Error finding container 027056d92ce45d4005f0e87defc317608d19ccdee532398a4de0e870d806d668: Status 404 returned error can't find the container with id 027056d92ce45d4005f0e87defc317608d19ccdee532398a4de0e870d806d668 Oct 30 22:33:59.062144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:59.062118 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8"] Oct 30 22:33:59.945175 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:59.945148 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8" event={"ID":"20af7c9c-26cb-4ae1-b572-8a1ed8525b28","Type":"ContainerStarted","Data":"ae47b57194803bee7a76210f05fd8404707c2b9bcdd8f84856a5049c7ee82230"} Oct 30 22:33:59.945540 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:33:59.945198 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8" event={"ID":"20af7c9c-26cb-4ae1-b572-8a1ed8525b28","Type":"ContainerStarted","Data":"027056d92ce45d4005f0e87defc317608d19ccdee532398a4de0e870d806d668"} Oct 30 22:34:00.062333 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:00.062285 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8" podStartSLOduration=2.06226738 podStartE2EDuration="2.06226738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:00.061888296 +0000 UTC m=+2973.162320028" watchObservedRunningTime="2026-10-30 22:34:00.06226738 +0000 UTC m=+2973.162699114" Oct 30 22:34:00.953970 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:00.953934 4335 generic.go:356] "Generic (PLEG): container finished" podID="20af7c9c-26cb-4ae1-b572-8a1ed8525b28" containerID="ae47b57194803bee7a76210f05fd8404707c2b9bcdd8f84856a5049c7ee82230" exitCode=0 Oct 30 22:34:00.954474 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:00.954026 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8" event={"ID":"20af7c9c-26cb-4ae1-b572-8a1ed8525b28","Type":"ContainerDied","Data":"ae47b57194803bee7a76210f05fd8404707c2b9bcdd8f84856a5049c7ee82230"} Oct 30 22:34:02.105375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:02.105352 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8" Oct 30 22:34:02.181324 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:02.181302 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-maas-api-xks-ca\") pod \"20af7c9c-26cb-4ae1-b572-8a1ed8525b28\" (UID: \"20af7c9c-26cb-4ae1-b572-8a1ed8525b28\") " Oct 30 22:34:02.181504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:02.181483 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-maas-api-openshift-ca\") pod \"20af7c9c-26cb-4ae1-b572-8a1ed8525b28\" (UID: \"20af7c9c-26cb-4ae1-b572-8a1ed8525b28\") " Oct 30 22:34:02.181592 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:02.181575 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-kube-api-access-ps65q\" (UniqueName: \"kubernetes.io/projected/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-kube-api-access-ps65q\") pod \"20af7c9c-26cb-4ae1-b572-8a1ed8525b28\" (UID: \"20af7c9c-26cb-4ae1-b572-8a1ed8525b28\") " Oct 30 22:34:02.181910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:02.181881 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-maas-api-openshift-ca" pod "20af7c9c-26cb-4ae1-b572-8a1ed8525b28" (UID: "20af7c9c-26cb-4ae1-b572-8a1ed8525b28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:02.182023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:02.181968 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:02.183655 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:02.183624 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-maas-api-xks-ca" pod "20af7c9c-26cb-4ae1-b572-8a1ed8525b28" (UID: "20af7c9c-26cb-4ae1-b572-8a1ed8525b28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:02.183812 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:02.183793 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-kube-api-access-ps65q" pod "20af7c9c-26cb-4ae1-b572-8a1ed8525b28" (UID: "20af7c9c-26cb-4ae1-b572-8a1ed8525b28"). InnerVolumeSpecName "kube-api-access-ps65q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:02.283576 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:02.283509 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps65q\" (UniqueName: \"kubernetes.io/projected/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-kube-api-access-ps65q\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:02.283576 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:02.283541 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20af7c9c-26cb-4ae1-b572-8a1ed8525b28-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:02.970516 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:02.970487 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8" Oct 30 22:34:02.970718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:02.970491 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8" event={"ID":"20af7c9c-26cb-4ae1-b572-8a1ed8525b28","Type":"ContainerDied","Data":"027056d92ce45d4005f0e87defc317608d19ccdee532398a4de0e870d806d668"} Oct 30 22:34:02.970718 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:02.970626 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="027056d92ce45d4005f0e87defc317608d19ccdee532398a4de0e870d806d668" Oct 30 22:34:07.229594 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:07.229554 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn"] Oct 30 22:34:07.336680 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:07.336629 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb"] Oct 30 22:34:12.231136 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.231092 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" podUID="7f63b65f-f924-4f71-afc5-3a5418226ffa" containerName="payload-processing" containerID="cri-o://acf6f6bb3d488e338e68d12861556465d1024cfa6a02a95009773f868217522e" gracePeriod=25 Oct 30 22:34:12.238897 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.238863 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" podUID="c5287819-aaee-4677-abb3-ee0f724b6d19" containerName="main" containerID="cri-o://7dd6af610e19290d5f1b1d54759aedd8fd8f5ff7d655dd3e3fa48a435e4a126f" gracePeriod=45 Oct 30 22:34:12.337763 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.337722 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" podUID="a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc" containerName="payload-pre-processing" containerID="cri-o://727be62bb9254a13fc445a4e23d9486e4efc46010c82cee5e8c29362216c6d5c" gracePeriod=25 Oct 30 22:34:12.510507 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.510484 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:34:12.592450 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.592424 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5287819-aaee-4677-abb3-ee0f724b6d19-kube-api-access-bjh2f\" (UniqueName: \"kubernetes.io/projected/c5287819-aaee-4677-abb3-ee0f724b6d19-kube-api-access-bjh2f\") pod \"c5287819-aaee-4677-abb3-ee0f724b6d19\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " Oct 30 22:34:12.592576 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.592473 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-dshm\") pod \"c5287819-aaee-4677-abb3-ee0f724b6d19\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " Oct 30 22:34:12.592576 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.592497 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-model-cache\") pod \"c5287819-aaee-4677-abb3-ee0f724b6d19\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " Oct 30 22:34:12.592576 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.592550 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5287819-aaee-4677-abb3-ee0f724b6d19-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5287819-aaee-4677-abb3-ee0f724b6d19-tls-certs\") pod \"c5287819-aaee-4677-abb3-ee0f724b6d19\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " Oct 30 22:34:12.592831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.592584 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-tmp-dir\") pod \"c5287819-aaee-4677-abb3-ee0f724b6d19\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " Oct 30 22:34:12.592831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.592677 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-home\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-home\") pod \"c5287819-aaee-4677-abb3-ee0f724b6d19\" (UID: \"c5287819-aaee-4677-abb3-ee0f724b6d19\") " Oct 30 22:34:12.592959 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.592834 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-model-cache" pod "c5287819-aaee-4677-abb3-ee0f724b6d19" (UID: "c5287819-aaee-4677-abb3-ee0f724b6d19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:12.593022 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.592948 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-tmp-dir" pod "c5287819-aaee-4677-abb3-ee0f724b6d19" (UID: "c5287819-aaee-4677-abb3-ee0f724b6d19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:12.593022 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.592961 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-home" pod "c5287819-aaee-4677-abb3-ee0f724b6d19" (UID: "c5287819-aaee-4677-abb3-ee0f724b6d19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:12.593305 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.593102 4335 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-model-cache\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:12.593305 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.593116 4335 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-tmp-dir\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:12.593305 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.593130 4335 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-home\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:12.594753 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.594728 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-dshm" pod "c5287819-aaee-4677-abb3-ee0f724b6d19" (UID: "c5287819-aaee-4677-abb3-ee0f724b6d19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:12.594871 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.594855 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5287819-aaee-4677-abb3-ee0f724b6d19-tls-certs" pod "c5287819-aaee-4677-abb3-ee0f724b6d19" (UID: "c5287819-aaee-4677-abb3-ee0f724b6d19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:12.594979 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.594965 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5287819-aaee-4677-abb3-ee0f724b6d19-kube-api-access-bjh2f" pod "c5287819-aaee-4677-abb3-ee0f724b6d19" (UID: "c5287819-aaee-4677-abb3-ee0f724b6d19"). InnerVolumeSpecName "kube-api-access-bjh2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:12.694732 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.694706 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjh2f\" (UniqueName: \"kubernetes.io/projected/c5287819-aaee-4677-abb3-ee0f724b6d19-kube-api-access-bjh2f\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:12.694846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.694737 4335 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5287819-aaee-4677-abb3-ee0f724b6d19-dshm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:12.694846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:12.694753 4335 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5287819-aaee-4677-abb3-ee0f724b6d19-tls-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:13.043108 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:13.043079 4335 generic.go:356] "Generic (PLEG): container finished" podID="c5287819-aaee-4677-abb3-ee0f724b6d19" containerID="7dd6af610e19290d5f1b1d54759aedd8fd8f5ff7d655dd3e3fa48a435e4a126f" exitCode=0 Oct 30 22:34:13.043262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:13.043124 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" Oct 30 22:34:13.043262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:13.043158 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" event={"ID":"c5287819-aaee-4677-abb3-ee0f724b6d19","Type":"ContainerDied","Data":"7dd6af610e19290d5f1b1d54759aedd8fd8f5ff7d655dd3e3fa48a435e4a126f"} Oct 30 22:34:13.043262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:13.043203 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" event={"ID":"c5287819-aaee-4677-abb3-ee0f724b6d19","Type":"ContainerDied","Data":"cddc4042bff80c8b0825a14a8804440fc4df862443d9e003a283db49fe71e820"} Oct 30 22:34:13.043262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:13.043221 4335 scope.go:122] "RemoveContainer" containerID="7dd6af610e19290d5f1b1d54759aedd8fd8f5ff7d655dd3e3fa48a435e4a126f" Oct 30 22:34:13.054049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:13.054033 4335 scope.go:122] "RemoveContainer" containerID="7dd6af610e19290d5f1b1d54759aedd8fd8f5ff7d655dd3e3fa48a435e4a126f" Oct 30 22:34:13.054304 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:34:13.054288 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd6af610e19290d5f1b1d54759aedd8fd8f5ff7d655dd3e3fa48a435e4a126f\": container with ID starting with 7dd6af610e19290d5f1b1d54759aedd8fd8f5ff7d655dd3e3fa48a435e4a126f not found: ID does not exist" containerID="7dd6af610e19290d5f1b1d54759aedd8fd8f5ff7d655dd3e3fa48a435e4a126f" Oct 30 22:34:13.054359 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:13.054311 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd6af610e19290d5f1b1d54759aedd8fd8f5ff7d655dd3e3fa48a435e4a126f"} err="failed to get container status \"7dd6af610e19290d5f1b1d54759aedd8fd8f5ff7d655dd3e3fa48a435e4a126f\": rpc error: code = NotFound desc = could not find container \"7dd6af610e19290d5f1b1d54759aedd8fd8f5ff7d655dd3e3fa48a435e4a126f\": container with ID starting with 7dd6af610e19290d5f1b1d54759aedd8fd8f5ff7d655dd3e3fa48a435e4a126f not found: ID does not exist" Oct 30 22:34:13.208401 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:13.208377 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv"] Oct 30 22:34:13.213328 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:13.213308 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv"] Oct 30 22:34:13.446924 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:13.446887 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-87b6b3/test-model-87b6b3-kserve-86957f4f79-kmbdv" podUID="c5287819-aaee-4677-abb3-ee0f724b6d19" containerName="main" probeResult="failure" output="Get \"https://10.133.1.7:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:34:15.062370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:15.062336 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5287819-aaee-4677-abb3-ee0f724b6d19" path="/var/lib/kubelet/pods/c5287819-aaee-4677-abb3-ee0f724b6d19/volumes" Oct 30 22:34:15.995844 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:15.995801 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" podUID="a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:15.996029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:15.995843 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" podUID="7f63b65f-f924-4f71-afc5-3a5418226ffa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:20.645737 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:20.645703 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:34:25.320048 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:34:25.320015 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:34:25.995979 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:25.995941 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" podUID="7f63b65f-f924-4f71-afc5-3a5418226ffa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:25.996816 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:25.996789 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" podUID="a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:28.424222 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:28.424187 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss"] Oct 30 22:34:28.424789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:28.424428 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" podUID="e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" containerName="istio-proxy" containerID="cri-o://2a21a39e8d02929f95dae2ccb10b46f41b54a8a8275d2d52918263a666839e16" gracePeriod=30 Oct 30 22:34:28.487234 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:28.487208 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:34:28.562875 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:28.562850 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:34:30.436271 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:30.436239 4335 scope.go:122] "RemoveContainer" containerID="2eedc53b98ef41988f0f02c643f1c4a05c214be813b7762316dc12d3f220aba2" Oct 30 22:34:30.446005 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:30.445984 4335 scope.go:122] "RemoveContainer" containerID="52afdd469c44597b336bd8a97d8584014a86f437b479fe04289297e020320d56" Oct 30 22:34:30.455274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:30.455256 4335 scope.go:122] "RemoveContainer" containerID="d2ddf45c8c30153b1e6800515015ebe71c539b07ecb5d4b31fe3514622dcc1f3" Oct 30 22:34:30.463780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:30.463760 4335 scope.go:122] "RemoveContainer" containerID="4384d9560dd1b427eb6fff46c07f7d9eb4b4e922a39e84cf3585cb396d2b9d18" Oct 30 22:34:30.823030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:30.823000 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs"] Oct 30 22:34:30.930347 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:30.930322 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn"] Oct 30 22:34:32.542982 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:32.542946 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" Oct 30 22:34:32.602845 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:32.602822 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" Oct 30 22:34:32.711521 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:32.711482 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc-plugins-config-volume\") pod \"a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc\" (UID: \"a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc\") " Oct 30 22:34:32.711707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:32.711606 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f63b65f-f924-4f71-afc5-3a5418226ffa-kube-api-access-trv6q\" (UniqueName: \"kubernetes.io/projected/7f63b65f-f924-4f71-afc5-3a5418226ffa-kube-api-access-trv6q\") pod \"7f63b65f-f924-4f71-afc5-3a5418226ffa\" (UID: \"7f63b65f-f924-4f71-afc5-3a5418226ffa\") " Oct 30 22:34:32.711707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:32.711661 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f63b65f-f924-4f71-afc5-3a5418226ffa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f63b65f-f924-4f71-afc5-3a5418226ffa-plugins-config-volume\") pod \"7f63b65f-f924-4f71-afc5-3a5418226ffa\" (UID: \"7f63b65f-f924-4f71-afc5-3a5418226ffa\") " Oct 30 22:34:32.711794 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:32.711703 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc-kube-api-access-cpm8j\" (UniqueName: \"kubernetes.io/projected/a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc-kube-api-access-cpm8j\") pod \"a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc\" (UID: \"a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc\") " Oct 30 22:34:32.712055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:32.712025 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc-plugins-config-volume" pod "a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc" (UID: "a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:32.712131 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:32.712054 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f63b65f-f924-4f71-afc5-3a5418226ffa-plugins-config-volume" pod "7f63b65f-f924-4f71-afc5-3a5418226ffa" (UID: "7f63b65f-f924-4f71-afc5-3a5418226ffa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:32.713954 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:32.713931 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f63b65f-f924-4f71-afc5-3a5418226ffa-kube-api-access-trv6q" pod "7f63b65f-f924-4f71-afc5-3a5418226ffa" (UID: "7f63b65f-f924-4f71-afc5-3a5418226ffa"). InnerVolumeSpecName "kube-api-access-trv6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:32.714012 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:32.713955 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc-kube-api-access-cpm8j" pod "a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc" (UID: "a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc"). InnerVolumeSpecName "kube-api-access-cpm8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:32.812566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:32.812530 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trv6q\" (UniqueName: \"kubernetes.io/projected/7f63b65f-f924-4f71-afc5-3a5418226ffa-kube-api-access-trv6q\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:32.812566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:32.812555 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f63b65f-f924-4f71-afc5-3a5418226ffa-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:32.812566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:32.812566 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpm8j\" (UniqueName: \"kubernetes.io/projected/a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc-kube-api-access-cpm8j\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:32.812566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:32.812575 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:33.185736 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.185705 4335 generic.go:356] "Generic (PLEG): container finished" podID="a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc" containerID="727be62bb9254a13fc445a4e23d9486e4efc46010c82cee5e8c29362216c6d5c" exitCode=0 Oct 30 22:34:33.185880 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.185750 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" Oct 30 22:34:33.185880 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.185794 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" event={"ID":"a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc","Type":"ContainerDied","Data":"727be62bb9254a13fc445a4e23d9486e4efc46010c82cee5e8c29362216c6d5c"} Oct 30 22:34:33.185880 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.185850 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb" event={"ID":"a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc","Type":"ContainerDied","Data":"b970fb011c2ff49915ae4a9a678fae6e4fe2e94cb44511154909f60d8bfffc1e"} Oct 30 22:34:33.186046 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.185883 4335 scope.go:122] "RemoveContainer" containerID="727be62bb9254a13fc445a4e23d9486e4efc46010c82cee5e8c29362216c6d5c" Oct 30 22:34:33.187684 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.187643 4335 generic.go:356] "Generic (PLEG): container finished" podID="7f63b65f-f924-4f71-afc5-3a5418226ffa" containerID="acf6f6bb3d488e338e68d12861556465d1024cfa6a02a95009773f868217522e" exitCode=0 Oct 30 22:34:33.187824 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.187699 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" Oct 30 22:34:33.187824 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.187750 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" event={"ID":"7f63b65f-f924-4f71-afc5-3a5418226ffa","Type":"ContainerDied","Data":"acf6f6bb3d488e338e68d12861556465d1024cfa6a02a95009773f868217522e"} Oct 30 22:34:33.187824 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.187795 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn" event={"ID":"7f63b65f-f924-4f71-afc5-3a5418226ffa","Type":"ContainerDied","Data":"480a0a45fc62620c3863e727563e1a0277cba87076ea224fa1376c0cfb961639"} Oct 30 22:34:33.195960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.195942 4335 scope.go:122] "RemoveContainer" containerID="727be62bb9254a13fc445a4e23d9486e4efc46010c82cee5e8c29362216c6d5c" Oct 30 22:34:33.196224 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:34:33.196202 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727be62bb9254a13fc445a4e23d9486e4efc46010c82cee5e8c29362216c6d5c\": container with ID starting with 727be62bb9254a13fc445a4e23d9486e4efc46010c82cee5e8c29362216c6d5c not found: ID does not exist" containerID="727be62bb9254a13fc445a4e23d9486e4efc46010c82cee5e8c29362216c6d5c" Oct 30 22:34:33.196276 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.196233 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727be62bb9254a13fc445a4e23d9486e4efc46010c82cee5e8c29362216c6d5c"} err="failed to get container status \"727be62bb9254a13fc445a4e23d9486e4efc46010c82cee5e8c29362216c6d5c\": rpc error: code = NotFound desc = could not find container \"727be62bb9254a13fc445a4e23d9486e4efc46010c82cee5e8c29362216c6d5c\": container with ID starting with 727be62bb9254a13fc445a4e23d9486e4efc46010c82cee5e8c29362216c6d5c not found: ID does not exist" Oct 30 22:34:33.196276 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.196251 4335 scope.go:122] "RemoveContainer" containerID="acf6f6bb3d488e338e68d12861556465d1024cfa6a02a95009773f868217522e" Oct 30 22:34:33.205537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.205514 4335 scope.go:122] "RemoveContainer" containerID="acf6f6bb3d488e338e68d12861556465d1024cfa6a02a95009773f868217522e" Oct 30 22:34:33.205859 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:34:33.205837 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acf6f6bb3d488e338e68d12861556465d1024cfa6a02a95009773f868217522e\": container with ID starting with acf6f6bb3d488e338e68d12861556465d1024cfa6a02a95009773f868217522e not found: ID does not exist" containerID="acf6f6bb3d488e338e68d12861556465d1024cfa6a02a95009773f868217522e" Oct 30 22:34:33.205975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.205862 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acf6f6bb3d488e338e68d12861556465d1024cfa6a02a95009773f868217522e"} err="failed to get container status \"acf6f6bb3d488e338e68d12861556465d1024cfa6a02a95009773f868217522e\": rpc error: code = NotFound desc = could not find container \"acf6f6bb3d488e338e68d12861556465d1024cfa6a02a95009773f868217522e\": container with ID starting with acf6f6bb3d488e338e68d12861556465d1024cfa6a02a95009773f868217522e not found: ID does not exist" Oct 30 22:34:33.353739 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.353683 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn"] Oct 30 22:34:33.360994 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.360969 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-026c11-5c9567ddc5-kxmbn"] Oct 30 22:34:33.474157 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.474135 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb"] Oct 30 22:34:33.477187 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.477166 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-026c11-7dd5f4b54c-bwggb"] Oct 30 22:34:33.689945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.689918 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:34:33.824027 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.824000 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-envoy\") pod \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " Oct 30 22:34:33.824201 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.824037 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-workload-certs\") pod \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " Oct 30 22:34:33.824201 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.824110 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-credential-socket\") pod \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " Oct 30 22:34:33.824201 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.824156 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-workload-socket\") pod \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " Oct 30 22:34:33.824386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.824207 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istiod-ca-cert\") pod \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " Oct 30 22:34:33.824386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.824248 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-podinfo\") pod \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " Oct 30 22:34:33.824386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.824280 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-kube-api-access-rtwkf\" (UniqueName: \"kubernetes.io/projected/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-kube-api-access-rtwkf\") pod \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " Oct 30 22:34:33.824386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.824341 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-token\" (UniqueName: \"kubernetes.io/projected/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-token\") pod \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " Oct 30 22:34:33.824386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.824369 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-data\") pod \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\" (UID: \"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae\") " Oct 30 22:34:33.824386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.824374 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-workload-certs" pod "e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" (UID: "e3b5d5ce-678d-4e48-87f7-5d6b2524ceae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:33.824751 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.824452 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-credential-socket" pod "e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" (UID: "e3b5d5ce-678d-4e48-87f7-5d6b2524ceae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:33.824751 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.824582 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-workload-socket" pod "e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" (UID: "e3b5d5ce-678d-4e48-87f7-5d6b2524ceae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:33.824864 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.824760 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-data" pod "e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" (UID: "e3b5d5ce-678d-4e48-87f7-5d6b2524ceae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:33.824864 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.824806 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istiod-ca-cert" pod "e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" (UID: "e3b5d5ce-678d-4e48-87f7-5d6b2524ceae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:33.824975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.824890 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:33.824975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.824910 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:33.824975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.824925 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:33.824975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.824939 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:33.826904 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.826882 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-kube-api-access-rtwkf" pod "e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" (UID: "e3b5d5ce-678d-4e48-87f7-5d6b2524ceae"). InnerVolumeSpecName "kube-api-access-rtwkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:33.827586 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.827556 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-token" pod "e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" (UID: "e3b5d5ce-678d-4e48-87f7-5d6b2524ceae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:33.828395 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.828373 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-envoy" pod "e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" (UID: "e3b5d5ce-678d-4e48-87f7-5d6b2524ceae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:33.828577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.828560 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-podinfo" pod "e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" (UID: "e3b5d5ce-678d-4e48-87f7-5d6b2524ceae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:33.926392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.926323 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:33.926392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.926350 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:33.926392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.926362 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:33.926392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.926374 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:33.926392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:33.926390 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtwkf\" (UniqueName: \"kubernetes.io/projected/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae-kube-api-access-rtwkf\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:34.196076 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:34.196001 4335 generic.go:356] "Generic (PLEG): container finished" podID="e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" containerID="2a21a39e8d02929f95dae2ccb10b46f41b54a8a8275d2d52918263a666839e16" exitCode=0 Oct 30 22:34:34.196076 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:34.196029 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" Oct 30 22:34:34.196280 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:34.196065 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" event={"ID":"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae","Type":"ContainerDied","Data":"2a21a39e8d02929f95dae2ccb10b46f41b54a8a8275d2d52918263a666839e16"} Oct 30 22:34:34.196280 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:34.196123 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss" event={"ID":"e3b5d5ce-678d-4e48-87f7-5d6b2524ceae","Type":"ContainerDied","Data":"e5d259b162f8b95239bba563bf5e5782de7902bd0a7f46aa6191cb8f34794436"} Oct 30 22:34:34.196280 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:34.196153 4335 scope.go:122] "RemoveContainer" containerID="2a21a39e8d02929f95dae2ccb10b46f41b54a8a8275d2d52918263a666839e16" Oct 30 22:34:34.207059 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:34.207040 4335 scope.go:122] "RemoveContainer" containerID="2a21a39e8d02929f95dae2ccb10b46f41b54a8a8275d2d52918263a666839e16" Oct 30 22:34:34.207326 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:34:34.207304 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a21a39e8d02929f95dae2ccb10b46f41b54a8a8275d2d52918263a666839e16\": container with ID starting with 2a21a39e8d02929f95dae2ccb10b46f41b54a8a8275d2d52918263a666839e16 not found: ID does not exist" containerID="2a21a39e8d02929f95dae2ccb10b46f41b54a8a8275d2d52918263a666839e16" Oct 30 22:34:34.207381 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:34.207333 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a21a39e8d02929f95dae2ccb10b46f41b54a8a8275d2d52918263a666839e16"} err="failed to get container status \"2a21a39e8d02929f95dae2ccb10b46f41b54a8a8275d2d52918263a666839e16\": rpc error: code = NotFound desc = could not find container \"2a21a39e8d02929f95dae2ccb10b46f41b54a8a8275d2d52918263a666839e16\": container with ID starting with 2a21a39e8d02929f95dae2ccb10b46f41b54a8a8275d2d52918263a666839e16 not found: ID does not exist" Oct 30 22:34:34.368710 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:34.368671 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss"] Oct 30 22:34:34.376384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:34.376359 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n6mss"] Oct 30 22:34:35.057234 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:35.057206 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f63b65f-f924-4f71-afc5-3a5418226ffa" path="/var/lib/kubelet/pods/7f63b65f-f924-4f71-afc5-3a5418226ffa/volumes" Oct 30 22:34:35.057648 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:35.057569 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc" path="/var/lib/kubelet/pods/a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc/volumes" Oct 30 22:34:35.057986 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:35.057972 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" path="/var/lib/kubelet/pods/e3b5d5ce-678d-4e48-87f7-5d6b2524ceae/volumes" Oct 30 22:34:35.824171 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:35.824136 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs" podUID="fc7b946c-d083-4b2f-829a-7066e9309ad5" containerName="payload-processing" containerID="cri-o://88e293f0bb280c74d04666eab9fdc4b900a62f91671592fc1913a1e6d9d70270" gracePeriod=25 Oct 30 22:34:35.931394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:35.931339 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" podUID="65723ec8-f97a-4172-86a5-9dfbffdea329" containerName="payload-pre-processing" containerID="cri-o://c37fdf8be6e2a77262b6a9845e60d0536718becfc12b4079f072010f6654b721" gracePeriod=25 Oct 30 22:34:36.082315 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:36.082259 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs" Oct 30 22:34:36.148195 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:36.148170 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc7b946c-d083-4b2f-829a-7066e9309ad5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7b946c-d083-4b2f-829a-7066e9309ad5-plugins-config-volume\") pod \"fc7b946c-d083-4b2f-829a-7066e9309ad5\" (UID: \"fc7b946c-d083-4b2f-829a-7066e9309ad5\") " Oct 30 22:34:36.148352 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:36.148225 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc7b946c-d083-4b2f-829a-7066e9309ad5-kube-api-access-ppnmr\" (UniqueName: \"kubernetes.io/projected/fc7b946c-d083-4b2f-829a-7066e9309ad5-kube-api-access-ppnmr\") pod \"fc7b946c-d083-4b2f-829a-7066e9309ad5\" (UID: \"fc7b946c-d083-4b2f-829a-7066e9309ad5\") " Oct 30 22:34:36.148590 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:36.148563 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc7b946c-d083-4b2f-829a-7066e9309ad5-plugins-config-volume" pod "fc7b946c-d083-4b2f-829a-7066e9309ad5" (UID: "fc7b946c-d083-4b2f-829a-7066e9309ad5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:36.148921 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:36.148878 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7b946c-d083-4b2f-829a-7066e9309ad5-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:36.150575 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:36.150553 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7b946c-d083-4b2f-829a-7066e9309ad5-kube-api-access-ppnmr" pod "fc7b946c-d083-4b2f-829a-7066e9309ad5" (UID: "fc7b946c-d083-4b2f-829a-7066e9309ad5"). InnerVolumeSpecName "kube-api-access-ppnmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:36.213699 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:36.213672 4335 generic.go:356] "Generic (PLEG): container finished" podID="fc7b946c-d083-4b2f-829a-7066e9309ad5" containerID="88e293f0bb280c74d04666eab9fdc4b900a62f91671592fc1913a1e6d9d70270" exitCode=0 Oct 30 22:34:36.213836 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:36.213716 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs" Oct 30 22:34:36.213836 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:36.213751 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs" event={"ID":"fc7b946c-d083-4b2f-829a-7066e9309ad5","Type":"ContainerDied","Data":"88e293f0bb280c74d04666eab9fdc4b900a62f91671592fc1913a1e6d9d70270"} Oct 30 22:34:36.213836 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:36.213808 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs" event={"ID":"fc7b946c-d083-4b2f-829a-7066e9309ad5","Type":"ContainerDied","Data":"44ab1c10a18b9fadb256fc6406cbc1b9bd085524ad1f5a74cb7e680e7a12d8c8"} Oct 30 22:34:36.213836 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:36.213830 4335 scope.go:122] "RemoveContainer" containerID="88e293f0bb280c74d04666eab9fdc4b900a62f91671592fc1913a1e6d9d70270" Oct 30 22:34:36.226818 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:36.226301 4335 scope.go:122] "RemoveContainer" containerID="88e293f0bb280c74d04666eab9fdc4b900a62f91671592fc1913a1e6d9d70270" Oct 30 22:34:36.226818 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:34:36.226609 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88e293f0bb280c74d04666eab9fdc4b900a62f91671592fc1913a1e6d9d70270\": container with ID starting with 88e293f0bb280c74d04666eab9fdc4b900a62f91671592fc1913a1e6d9d70270 not found: ID does not exist" containerID="88e293f0bb280c74d04666eab9fdc4b900a62f91671592fc1913a1e6d9d70270" Oct 30 22:34:36.226818 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:36.226653 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e293f0bb280c74d04666eab9fdc4b900a62f91671592fc1913a1e6d9d70270"} err="failed to get container status \"88e293f0bb280c74d04666eab9fdc4b900a62f91671592fc1913a1e6d9d70270\": rpc error: code = NotFound desc = could not find container \"88e293f0bb280c74d04666eab9fdc4b900a62f91671592fc1913a1e6d9d70270\": container with ID starting with 88e293f0bb280c74d04666eab9fdc4b900a62f91671592fc1913a1e6d9d70270 not found: ID does not exist" Oct 30 22:34:36.250023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:36.249991 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppnmr\" (UniqueName: \"kubernetes.io/projected/fc7b946c-d083-4b2f-829a-7066e9309ad5-kube-api-access-ppnmr\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:36.393583 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:36.393513 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs"] Oct 30 22:34:36.398956 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:36.398935 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-87b6b3-6bc886f97-l49vs"] Oct 30 22:34:37.057652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:37.057609 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc7b946c-d083-4b2f-829a-7066e9309ad5" path="/var/lib/kubelet/pods/fc7b946c-d083-4b2f-829a-7066e9309ad5/volumes" Oct 30 22:34:40.045415 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:40.045343 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:34:45.177607 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:45.177575 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" podUID="65723ec8-f97a-4172-86a5-9dfbffdea329" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:55.177740 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:55.177706 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" podUID="65723ec8-f97a-4172-86a5-9dfbffdea329" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:34:55.321367 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:34:55.321326 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:34:56.093951 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:56.093915 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c"] Oct 30 22:34:56.094203 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:56.094178 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" podUID="f2ad9d54-24dc-40ea-9397-5bfc51b9d786" containerName="istio-proxy" containerID="cri-o://1a7dbe29556bde2ef478f56041ac7a7733557d2b05ed7a749ae5acf9707af41e" gracePeriod=30 Oct 30 22:34:56.215890 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:56.215859 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" Oct 30 22:34:56.333947 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:56.333913 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65723ec8-f97a-4172-86a5-9dfbffdea329-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65723ec8-f97a-4172-86a5-9dfbffdea329-plugins-config-volume\") pod \"65723ec8-f97a-4172-86a5-9dfbffdea329\" (UID: \"65723ec8-f97a-4172-86a5-9dfbffdea329\") " Oct 30 22:34:56.334108 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:56.334015 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65723ec8-f97a-4172-86a5-9dfbffdea329-kube-api-access-fxbdg\" (UniqueName: \"kubernetes.io/projected/65723ec8-f97a-4172-86a5-9dfbffdea329-kube-api-access-fxbdg\") pod \"65723ec8-f97a-4172-86a5-9dfbffdea329\" (UID: \"65723ec8-f97a-4172-86a5-9dfbffdea329\") " Oct 30 22:34:56.334372 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:56.334333 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65723ec8-f97a-4172-86a5-9dfbffdea329-plugins-config-volume" pod "65723ec8-f97a-4172-86a5-9dfbffdea329" (UID: "65723ec8-f97a-4172-86a5-9dfbffdea329"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:56.336210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:56.336189 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65723ec8-f97a-4172-86a5-9dfbffdea329-kube-api-access-fxbdg" pod "65723ec8-f97a-4172-86a5-9dfbffdea329" (UID: "65723ec8-f97a-4172-86a5-9dfbffdea329"). InnerVolumeSpecName "kube-api-access-fxbdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:56.356241 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:56.356218 4335 generic.go:356] "Generic (PLEG): container finished" podID="65723ec8-f97a-4172-86a5-9dfbffdea329" containerID="c37fdf8be6e2a77262b6a9845e60d0536718becfc12b4079f072010f6654b721" exitCode=0 Oct 30 22:34:56.356334 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:56.356263 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" Oct 30 22:34:56.356334 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:56.356299 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" event={"ID":"65723ec8-f97a-4172-86a5-9dfbffdea329","Type":"ContainerDied","Data":"c37fdf8be6e2a77262b6a9845e60d0536718becfc12b4079f072010f6654b721"} Oct 30 22:34:56.356434 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:56.356346 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn" event={"ID":"65723ec8-f97a-4172-86a5-9dfbffdea329","Type":"ContainerDied","Data":"726cfc60484bc01011cce50c51c1023b240783b449d676a396a339640909310c"} Oct 30 22:34:56.356434 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:56.356369 4335 scope.go:122] "RemoveContainer" containerID="c37fdf8be6e2a77262b6a9845e60d0536718becfc12b4079f072010f6654b721" Oct 30 22:34:56.366468 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:56.366451 4335 scope.go:122] "RemoveContainer" containerID="c37fdf8be6e2a77262b6a9845e60d0536718becfc12b4079f072010f6654b721" Oct 30 22:34:56.366759 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:34:56.366741 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37fdf8be6e2a77262b6a9845e60d0536718becfc12b4079f072010f6654b721\": container with ID starting with c37fdf8be6e2a77262b6a9845e60d0536718becfc12b4079f072010f6654b721 not found: ID does not exist" containerID="c37fdf8be6e2a77262b6a9845e60d0536718becfc12b4079f072010f6654b721" Oct 30 22:34:56.366802 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:56.366766 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37fdf8be6e2a77262b6a9845e60d0536718becfc12b4079f072010f6654b721"} err="failed to get container status \"c37fdf8be6e2a77262b6a9845e60d0536718becfc12b4079f072010f6654b721\": rpc error: code = NotFound desc = could not find container \"c37fdf8be6e2a77262b6a9845e60d0536718becfc12b4079f072010f6654b721\": container with ID starting with c37fdf8be6e2a77262b6a9845e60d0536718becfc12b4079f072010f6654b721 not found: ID does not exist" Oct 30 22:34:56.435370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:56.435342 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxbdg\" (UniqueName: \"kubernetes.io/projected/65723ec8-f97a-4172-86a5-9dfbffdea329-kube-api-access-fxbdg\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:56.435370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:56.435366 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65723ec8-f97a-4172-86a5-9dfbffdea329-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:56.491011 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:56.490956 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn"] Oct 30 22:34:56.494818 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:56.494786 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-87b6b3-85d8db6dc5-8w6nn"] Oct 30 22:34:57.076509 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:57.076446 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65723ec8-f97a-4172-86a5-9dfbffdea329" path="/var/lib/kubelet/pods/65723ec8-f97a-4172-86a5-9dfbffdea329/volumes" Oct 30 22:34:57.246142 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:57.246113 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:34:58.050338 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.050308 4335 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" secret="" err="secret \"e2e-inf-a-026c11-openshift-default-dockercfg-wzj69\" not found" Oct 30 22:34:58.053916 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.053882 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4"] Oct 30 22:34:58.371418 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.371336 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" podUID="1657e3aa-0cfe-49e5-b8ce-21888a334461" containerName="istio-proxy" containerID="cri-o://002618aed2cde0b287623624cd9bb75fd00ab8f7a9da4b7e07cc4ec5ac1d1d3e" gracePeriod=30 Oct 30 22:34:58.896236 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896202 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn"] Oct 30 22:34:58.896732 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896717 4335 cpu_manager.go:414] "containerMap: removing container" podUID="e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" containerName="istio-proxy" Oct 30 22:34:58.896732 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896733 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" containerName="istio-proxy" Oct 30 22:34:58.896846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896745 4335 cpu_manager.go:414] "containerMap: removing container" podUID="fc7b946c-d083-4b2f-829a-7066e9309ad5" containerName="payload-processing" Oct 30 22:34:58.896846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896751 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc7b946c-d083-4b2f-829a-7066e9309ad5" containerName="payload-processing" Oct 30 22:34:58.896846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896766 4335 cpu_manager.go:414] "containerMap: removing container" podUID="20af7c9c-26cb-4ae1-b572-8a1ed8525b28" containerName="revoke-keys" Oct 30 22:34:58.896846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896771 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20af7c9c-26cb-4ae1-b572-8a1ed8525b28" containerName="revoke-keys" Oct 30 22:34:58.896846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896792 4335 cpu_manager.go:414] "containerMap: removing container" podUID="c5287819-aaee-4677-abb3-ee0f724b6d19" containerName="main" Oct 30 22:34:58.896846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896797 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5287819-aaee-4677-abb3-ee0f724b6d19" containerName="main" Oct 30 22:34:58.896846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896804 4335 cpu_manager.go:414] "containerMap: removing container" podUID="65723ec8-f97a-4172-86a5-9dfbffdea329" containerName="payload-pre-processing" Oct 30 22:34:58.896846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896809 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65723ec8-f97a-4172-86a5-9dfbffdea329" containerName="payload-pre-processing" Oct 30 22:34:58.896846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896819 4335 cpu_manager.go:414] "containerMap: removing container" podUID="7f63b65f-f924-4f71-afc5-3a5418226ffa" containerName="payload-processing" Oct 30 22:34:58.896846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896824 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f63b65f-f924-4f71-afc5-3a5418226ffa" containerName="payload-processing" Oct 30 22:34:58.896846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896834 4335 cpu_manager.go:414] "containerMap: removing container" podUID="a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc" containerName="payload-pre-processing" Oct 30 22:34:58.896846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896839 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc" containerName="payload-pre-processing" Oct 30 22:34:58.897218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896909 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b5d5ce-678d-4e48-87f7-5d6b2524ceae" containerName="istio-proxy" Oct 30 22:34:58.897218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896915 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc7b946c-d083-4b2f-829a-7066e9309ad5" containerName="payload-processing" Oct 30 22:34:58.897218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896924 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="20af7c9c-26cb-4ae1-b572-8a1ed8525b28" containerName="revoke-keys" Oct 30 22:34:58.897218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896932 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5287819-aaee-4677-abb3-ee0f724b6d19" containerName="main" Oct 30 22:34:58.897218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896938 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="65723ec8-f97a-4172-86a5-9dfbffdea329" containerName="payload-pre-processing" Oct 30 22:34:58.897218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896944 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f63b65f-f924-4f71-afc5-3a5418226ffa" containerName="payload-processing" Oct 30 22:34:58.897218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.896950 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d790c0-2e37-43f7-ad4a-1a5c8b2637bc" containerName="payload-pre-processing" Oct 30 22:34:58.899996 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.899979 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn" Oct 30 22:34:58.953350 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.953296 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:34:58.953897 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.953872 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:34:58.954064 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.954048 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:34:58.958290 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.958266 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn"] Oct 30 22:34:58.970436 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.970412 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28zcg\" (UniqueName: \"kubernetes.io/projected/205109aa-7cf4-433c-8657-6c3174229ef4-kube-api-access-28zcg\") pod \"maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn\" (UID: \"205109aa-7cf4-433c-8657-6c3174229ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn" Oct 30 22:34:58.970553 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.970449 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/205109aa-7cf4-433c-8657-6c3174229ef4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn\" (UID: \"205109aa-7cf4-433c-8657-6c3174229ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn" Oct 30 22:34:58.970654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:58.970581 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/205109aa-7cf4-433c-8657-6c3174229ef4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn\" (UID: \"205109aa-7cf4-433c-8657-6c3174229ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn" Oct 30 22:34:59.002465 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:59.002449 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:34:59.071487 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:59.071458 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28zcg\" (UniqueName: \"kubernetes.io/projected/205109aa-7cf4-433c-8657-6c3174229ef4-kube-api-access-28zcg\") pod \"maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn\" (UID: \"205109aa-7cf4-433c-8657-6c3174229ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn" Oct 30 22:34:59.071646 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:59.071516 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/205109aa-7cf4-433c-8657-6c3174229ef4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn\" (UID: \"205109aa-7cf4-433c-8657-6c3174229ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn" Oct 30 22:34:59.071646 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:59.071634 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/205109aa-7cf4-433c-8657-6c3174229ef4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn\" (UID: \"205109aa-7cf4-433c-8657-6c3174229ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn" Oct 30 22:34:59.072264 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:59.072244 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/205109aa-7cf4-433c-8657-6c3174229ef4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn\" (UID: \"205109aa-7cf4-433c-8657-6c3174229ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn" Oct 30 22:34:59.074171 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:59.074146 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/205109aa-7cf4-433c-8657-6c3174229ef4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn\" (UID: \"205109aa-7cf4-433c-8657-6c3174229ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn" Oct 30 22:34:59.131799 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:59.131774 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28zcg\" (UniqueName: \"kubernetes.io/projected/205109aa-7cf4-433c-8657-6c3174229ef4-kube-api-access-28zcg\") pod \"maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn\" (UID: \"205109aa-7cf4-433c-8657-6c3174229ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn" Oct 30 22:34:59.212205 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:59.212153 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn" Oct 30 22:34:59.591428 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:59.591398 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn"] Oct 30 22:34:59.592477 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:34:59.592452 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205109aa_7cf4_433c_8657_6c3174229ef4.slice/crio-d0725e185bf03b0bc5550155da760acbf9d07f065188f3ad9fe3082b58dbc924 WatchSource:0}: Error finding container d0725e185bf03b0bc5550155da760acbf9d07f065188f3ad9fe3082b58dbc924: Status 404 returned error can't find the container with id d0725e185bf03b0bc5550155da760acbf9d07f065188f3ad9fe3082b58dbc924 Oct 30 22:34:59.652232 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:59.652201 4335 patch_prober.go:29] interesting pod/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:34:59.652349 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:34:59.652260 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" podUID="1657e3aa-0cfe-49e5-b8ce-21888a334461" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:35:00.385336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:00.385303 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn" event={"ID":"205109aa-7cf4-433c-8657-6c3174229ef4","Type":"ContainerStarted","Data":"b04c4a388808325c1b38d1506c193274f4cfd5e077292a51660470a539c42968"} Oct 30 22:35:00.385494 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:00.385351 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn" event={"ID":"205109aa-7cf4-433c-8657-6c3174229ef4","Type":"ContainerStarted","Data":"d0725e185bf03b0bc5550155da760acbf9d07f065188f3ad9fe3082b58dbc924"} Oct 30 22:35:00.503219 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:00.503143 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn" podStartSLOduration=2.5031271569999998 podStartE2EDuration="2.503127157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:00.500193688 +0000 UTC m=+3033.600625421" watchObservedRunningTime="2026-10-30 22:35:00.503127157 +0000 UTC m=+3033.603558869" Oct 30 22:35:01.353609 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.353581 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:35:01.393190 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.393133 4335 generic.go:356] "Generic (PLEG): container finished" podID="205109aa-7cf4-433c-8657-6c3174229ef4" containerID="b04c4a388808325c1b38d1506c193274f4cfd5e077292a51660470a539c42968" exitCode=0 Oct 30 22:35:01.393304 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.393215 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn" event={"ID":"205109aa-7cf4-433c-8657-6c3174229ef4","Type":"ContainerDied","Data":"b04c4a388808325c1b38d1506c193274f4cfd5e077292a51660470a539c42968"} Oct 30 22:35:01.395039 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.395016 4335 generic.go:356] "Generic (PLEG): container finished" podID="f2ad9d54-24dc-40ea-9397-5bfc51b9d786" containerID="1a7dbe29556bde2ef478f56041ac7a7733557d2b05ed7a749ae5acf9707af41e" exitCode=0 Oct 30 22:35:01.395138 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.395053 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" Oct 30 22:35:01.395138 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.395102 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" event={"ID":"f2ad9d54-24dc-40ea-9397-5bfc51b9d786","Type":"ContainerDied","Data":"1a7dbe29556bde2ef478f56041ac7a7733557d2b05ed7a749ae5acf9707af41e"} Oct 30 22:35:01.395236 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.395162 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c" event={"ID":"f2ad9d54-24dc-40ea-9397-5bfc51b9d786","Type":"ContainerDied","Data":"833b66cbd97147912ab34d3a90bae71c4bd1d9ccc4ac6a043be789bc70f7a904"} Oct 30 22:35:01.395236 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.395187 4335 scope.go:122] "RemoveContainer" containerID="1a7dbe29556bde2ef478f56041ac7a7733557d2b05ed7a749ae5acf9707af41e" Oct 30 22:35:01.404721 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.404705 4335 scope.go:122] "RemoveContainer" containerID="8cba0bc44621f7c2cec491e32470b93bc6487a4773f2cf7f0fd490b4ead19284" Oct 30 22:35:01.413576 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.413560 4335 scope.go:122] "RemoveContainer" containerID="1a7dbe29556bde2ef478f56041ac7a7733557d2b05ed7a749ae5acf9707af41e" Oct 30 22:35:01.413870 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:35:01.413852 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a7dbe29556bde2ef478f56041ac7a7733557d2b05ed7a749ae5acf9707af41e\": container with ID starting with 1a7dbe29556bde2ef478f56041ac7a7733557d2b05ed7a749ae5acf9707af41e not found: ID does not exist" containerID="1a7dbe29556bde2ef478f56041ac7a7733557d2b05ed7a749ae5acf9707af41e" Oct 30 22:35:01.413920 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.413878 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a7dbe29556bde2ef478f56041ac7a7733557d2b05ed7a749ae5acf9707af41e"} err="failed to get container status \"1a7dbe29556bde2ef478f56041ac7a7733557d2b05ed7a749ae5acf9707af41e\": rpc error: code = NotFound desc = could not find container \"1a7dbe29556bde2ef478f56041ac7a7733557d2b05ed7a749ae5acf9707af41e\": container with ID starting with 1a7dbe29556bde2ef478f56041ac7a7733557d2b05ed7a749ae5acf9707af41e not found: ID does not exist" Oct 30 22:35:01.413920 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.413896 4335 scope.go:122] "RemoveContainer" containerID="8cba0bc44621f7c2cec491e32470b93bc6487a4773f2cf7f0fd490b4ead19284" Oct 30 22:35:01.414133 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:35:01.414115 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cba0bc44621f7c2cec491e32470b93bc6487a4773f2cf7f0fd490b4ead19284\": container with ID starting with 8cba0bc44621f7c2cec491e32470b93bc6487a4773f2cf7f0fd490b4ead19284 not found: ID does not exist" containerID="8cba0bc44621f7c2cec491e32470b93bc6487a4773f2cf7f0fd490b4ead19284" Oct 30 22:35:01.414186 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.414139 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cba0bc44621f7c2cec491e32470b93bc6487a4773f2cf7f0fd490b4ead19284"} err="failed to get container status \"8cba0bc44621f7c2cec491e32470b93bc6487a4773f2cf7f0fd490b4ead19284\": rpc error: code = NotFound desc = could not find container \"8cba0bc44621f7c2cec491e32470b93bc6487a4773f2cf7f0fd490b4ead19284\": container with ID starting with 8cba0bc44621f7c2cec491e32470b93bc6487a4773f2cf7f0fd490b4ead19284 not found: ID does not exist" Oct 30 22:35:01.496226 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.496200 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-token\" (UniqueName: \"kubernetes.io/projected/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-token\") pod \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " Oct 30 22:35:01.496364 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.496244 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-workload-socket\") pod \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " Oct 30 22:35:01.496405 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.496386 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-envoy\") pod \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " Oct 30 22:35:01.496570 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.496527 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-workload-socket" pod "f2ad9d54-24dc-40ea-9397-5bfc51b9d786" (UID: "f2ad9d54-24dc-40ea-9397-5bfc51b9d786"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:01.500530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.496459 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-kube-api-access-c9w5g\" (UniqueName: \"kubernetes.io/projected/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-kube-api-access-c9w5g\") pod \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " Oct 30 22:35:01.500530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.497129 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istiod-ca-cert\") pod \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " Oct 30 22:35:01.500530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.497215 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-workload-certs\") pod \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " Oct 30 22:35:01.500530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.497295 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-data\") pod \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " Oct 30 22:35:01.500530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.497347 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-credential-socket\") pod \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " Oct 30 22:35:01.500530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.497692 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-credential-socket" pod "f2ad9d54-24dc-40ea-9397-5bfc51b9d786" (UID: "f2ad9d54-24dc-40ea-9397-5bfc51b9d786"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:01.500530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.498001 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istiod-ca-cert" pod "f2ad9d54-24dc-40ea-9397-5bfc51b9d786" (UID: "f2ad9d54-24dc-40ea-9397-5bfc51b9d786"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:01.500530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.498225 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-workload-certs" pod "f2ad9d54-24dc-40ea-9397-5bfc51b9d786" (UID: "f2ad9d54-24dc-40ea-9397-5bfc51b9d786"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:01.500530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.498386 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-data" pod "f2ad9d54-24dc-40ea-9397-5bfc51b9d786" (UID: "f2ad9d54-24dc-40ea-9397-5bfc51b9d786"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:01.500530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.499642 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-token" pod "f2ad9d54-24dc-40ea-9397-5bfc51b9d786" (UID: "f2ad9d54-24dc-40ea-9397-5bfc51b9d786"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:01.500530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.500026 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-envoy" pod "f2ad9d54-24dc-40ea-9397-5bfc51b9d786" (UID: "f2ad9d54-24dc-40ea-9397-5bfc51b9d786"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:01.500530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.500183 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-podinfo\") pod \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\" (UID: \"f2ad9d54-24dc-40ea-9397-5bfc51b9d786\") " Oct 30 22:35:01.501266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.500898 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-kube-api-access-c9w5g" pod "f2ad9d54-24dc-40ea-9397-5bfc51b9d786" (UID: "f2ad9d54-24dc-40ea-9397-5bfc51b9d786"). InnerVolumeSpecName "kube-api-access-c9w5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:01.501266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.500929 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:01.501266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.500961 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:01.501266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.500980 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:01.501266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.500993 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:01.501266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.501007 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:01.501266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.501171 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:01.501266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.501188 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:01.504350 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.503427 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-podinfo" pod "f2ad9d54-24dc-40ea-9397-5bfc51b9d786" (UID: "f2ad9d54-24dc-40ea-9397-5bfc51b9d786"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:35:01.601913 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.601889 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:01.602041 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.601914 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9w5g\" (UniqueName: \"kubernetes.io/projected/f2ad9d54-24dc-40ea-9397-5bfc51b9d786-kube-api-access-c9w5g\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:01.817249 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.817215 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c"] Oct 30 22:35:01.822493 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:01.822470 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-87b6b3-openshift-default-8c786574c-ph69c"] Oct 30 22:35:02.539231 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:02.539209 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn" Oct 30 22:35:02.611535 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:02.611501 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/205109aa-7cf4-433c-8657-6c3174229ef4-kube-api-access-28zcg\" (UniqueName: \"kubernetes.io/projected/205109aa-7cf4-433c-8657-6c3174229ef4-kube-api-access-28zcg\") pod \"205109aa-7cf4-433c-8657-6c3174229ef4\" (UID: \"205109aa-7cf4-433c-8657-6c3174229ef4\") " Oct 30 22:35:02.611712 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:02.611556 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/205109aa-7cf4-433c-8657-6c3174229ef4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/205109aa-7cf4-433c-8657-6c3174229ef4-maas-api-xks-ca\") pod \"205109aa-7cf4-433c-8657-6c3174229ef4\" (UID: \"205109aa-7cf4-433c-8657-6c3174229ef4\") " Oct 30 22:35:02.611712 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:02.611677 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/205109aa-7cf4-433c-8657-6c3174229ef4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/205109aa-7cf4-433c-8657-6c3174229ef4-maas-api-openshift-ca\") pod \"205109aa-7cf4-433c-8657-6c3174229ef4\" (UID: \"205109aa-7cf4-433c-8657-6c3174229ef4\") " Oct 30 22:35:02.612087 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:02.612055 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/205109aa-7cf4-433c-8657-6c3174229ef4-maas-api-openshift-ca" pod "205109aa-7cf4-433c-8657-6c3174229ef4" (UID: "205109aa-7cf4-433c-8657-6c3174229ef4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:02.613659 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:02.613638 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/205109aa-7cf4-433c-8657-6c3174229ef4-maas-api-xks-ca" pod "205109aa-7cf4-433c-8657-6c3174229ef4" (UID: "205109aa-7cf4-433c-8657-6c3174229ef4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:02.613806 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:02.613788 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205109aa-7cf4-433c-8657-6c3174229ef4-kube-api-access-28zcg" pod "205109aa-7cf4-433c-8657-6c3174229ef4" (UID: "205109aa-7cf4-433c-8657-6c3174229ef4"). InnerVolumeSpecName "kube-api-access-28zcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:02.713348 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:02.713286 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28zcg\" (UniqueName: \"kubernetes.io/projected/205109aa-7cf4-433c-8657-6c3174229ef4-kube-api-access-28zcg\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:02.713348 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:02.713311 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/205109aa-7cf4-433c-8657-6c3174229ef4-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:02.713348 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:02.713321 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/205109aa-7cf4-433c-8657-6c3174229ef4-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:03.056813 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.056787 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ad9d54-24dc-40ea-9397-5bfc51b9d786" path="/var/lib/kubelet/pods/f2ad9d54-24dc-40ea-9397-5bfc51b9d786/volumes" Oct 30 22:35:03.410690 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.410667 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn" Oct 30 22:35:03.410835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.410683 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn" event={"ID":"205109aa-7cf4-433c-8657-6c3174229ef4","Type":"ContainerDied","Data":"d0725e185bf03b0bc5550155da760acbf9d07f065188f3ad9fe3082b58dbc924"} Oct 30 22:35:03.410835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.410727 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0725e185bf03b0bc5550155da760acbf9d07f065188f3ad9fe3082b58dbc924" Oct 30 22:35:03.634129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.634107 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:35:03.724720 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.724664 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-credential-socket\") pod \"1657e3aa-0cfe-49e5-b8ce-21888a334461\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " Oct 30 22:35:03.724720 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.724701 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1657e3aa-0cfe-49e5-b8ce-21888a334461-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1657e3aa-0cfe-49e5-b8ce-21888a334461-istiod-ca-cert\") pod \"1657e3aa-0cfe-49e5-b8ce-21888a334461\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " Oct 30 22:35:03.724884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.724758 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-workload-socket\") pod \"1657e3aa-0cfe-49e5-b8ce-21888a334461\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " Oct 30 22:35:03.724884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.724780 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-envoy\") pod \"1657e3aa-0cfe-49e5-b8ce-21888a334461\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " Oct 30 22:35:03.724884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.724845 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1657e3aa-0cfe-49e5-b8ce-21888a334461-kube-api-access-fcqjf\" (UniqueName: \"kubernetes.io/projected/1657e3aa-0cfe-49e5-b8ce-21888a334461-kube-api-access-fcqjf\") pod \"1657e3aa-0cfe-49e5-b8ce-21888a334461\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " Oct 30 22:35:03.725055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.724890 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-credential-socket" pod "1657e3aa-0cfe-49e5-b8ce-21888a334461" (UID: "1657e3aa-0cfe-49e5-b8ce-21888a334461"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:03.725055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.724900 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-data\") pod \"1657e3aa-0cfe-49e5-b8ce-21888a334461\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " Oct 30 22:35:03.725055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.724975 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-workload-certs\") pod \"1657e3aa-0cfe-49e5-b8ce-21888a334461\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " Oct 30 22:35:03.725055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.724994 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-workload-socket" pod "1657e3aa-0cfe-49e5-b8ce-21888a334461" (UID: "1657e3aa-0cfe-49e5-b8ce-21888a334461"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:03.725055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.725014 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-token\" (UniqueName: \"kubernetes.io/projected/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-token\") pod \"1657e3aa-0cfe-49e5-b8ce-21888a334461\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " Oct 30 22:35:03.725348 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.725061 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-podinfo\") pod \"1657e3aa-0cfe-49e5-b8ce-21888a334461\" (UID: \"1657e3aa-0cfe-49e5-b8ce-21888a334461\") " Oct 30 22:35:03.725348 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.725189 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-data" pod "1657e3aa-0cfe-49e5-b8ce-21888a334461" (UID: "1657e3aa-0cfe-49e5-b8ce-21888a334461"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:03.725348 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.725246 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-workload-certs" pod "1657e3aa-0cfe-49e5-b8ce-21888a334461" (UID: "1657e3aa-0cfe-49e5-b8ce-21888a334461"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:03.725348 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.725248 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1657e3aa-0cfe-49e5-b8ce-21888a334461-istiod-ca-cert" pod "1657e3aa-0cfe-49e5-b8ce-21888a334461" (UID: "1657e3aa-0cfe-49e5-b8ce-21888a334461"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:03.725760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.725499 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:03.725760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.725523 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:03.725760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.725538 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:03.725760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.725633 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1657e3aa-0cfe-49e5-b8ce-21888a334461-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:03.725760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.725743 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:03.727221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.727193 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1657e3aa-0cfe-49e5-b8ce-21888a334461-kube-api-access-fcqjf" pod "1657e3aa-0cfe-49e5-b8ce-21888a334461" (UID: "1657e3aa-0cfe-49e5-b8ce-21888a334461"). InnerVolumeSpecName "kube-api-access-fcqjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:03.727334 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.727245 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-podinfo" pod "1657e3aa-0cfe-49e5-b8ce-21888a334461" (UID: "1657e3aa-0cfe-49e5-b8ce-21888a334461"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:35:03.727334 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.727253 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-token" pod "1657e3aa-0cfe-49e5-b8ce-21888a334461" (UID: "1657e3aa-0cfe-49e5-b8ce-21888a334461"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:03.727334 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.727282 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-envoy" pod "1657e3aa-0cfe-49e5-b8ce-21888a334461" (UID: "1657e3aa-0cfe-49e5-b8ce-21888a334461"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:03.826682 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.826658 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:03.826682 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.826681 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcqjf\" (UniqueName: \"kubernetes.io/projected/1657e3aa-0cfe-49e5-b8ce-21888a334461-kube-api-access-fcqjf\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:03.826886 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.826691 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:03.826886 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:03.826701 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1657e3aa-0cfe-49e5-b8ce-21888a334461-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:04.418905 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:04.418874 4335 generic.go:356] "Generic (PLEG): container finished" podID="1657e3aa-0cfe-49e5-b8ce-21888a334461" containerID="002618aed2cde0b287623624cd9bb75fd00ab8f7a9da4b7e07cc4ec5ac1d1d3e" exitCode=0 Oct 30 22:35:04.419084 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:04.418917 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" Oct 30 22:35:04.419084 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:04.418962 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" event={"ID":"1657e3aa-0cfe-49e5-b8ce-21888a334461","Type":"ContainerDied","Data":"002618aed2cde0b287623624cd9bb75fd00ab8f7a9da4b7e07cc4ec5ac1d1d3e"} Oct 30 22:35:04.419084 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:04.419027 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4" event={"ID":"1657e3aa-0cfe-49e5-b8ce-21888a334461","Type":"ContainerDied","Data":"c3c4b0265a0d2bae5c1fde466e04b239d87f1230a9634004ae6678468745408d"} Oct 30 22:35:04.419084 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:04.419055 4335 scope.go:122] "RemoveContainer" containerID="002618aed2cde0b287623624cd9bb75fd00ab8f7a9da4b7e07cc4ec5ac1d1d3e" Oct 30 22:35:04.432399 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:04.432376 4335 scope.go:122] "RemoveContainer" containerID="f305f04e50ee079295e625005f01f05fef22490b4cc3c54a926f557a040df7b4" Oct 30 22:35:04.441475 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:04.441456 4335 scope.go:122] "RemoveContainer" containerID="002618aed2cde0b287623624cd9bb75fd00ab8f7a9da4b7e07cc4ec5ac1d1d3e" Oct 30 22:35:04.441779 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:35:04.441755 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002618aed2cde0b287623624cd9bb75fd00ab8f7a9da4b7e07cc4ec5ac1d1d3e\": container with ID starting with 002618aed2cde0b287623624cd9bb75fd00ab8f7a9da4b7e07cc4ec5ac1d1d3e not found: ID does not exist" containerID="002618aed2cde0b287623624cd9bb75fd00ab8f7a9da4b7e07cc4ec5ac1d1d3e" Oct 30 22:35:04.441881 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:04.441788 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002618aed2cde0b287623624cd9bb75fd00ab8f7a9da4b7e07cc4ec5ac1d1d3e"} err="failed to get container status \"002618aed2cde0b287623624cd9bb75fd00ab8f7a9da4b7e07cc4ec5ac1d1d3e\": rpc error: code = NotFound desc = could not find container \"002618aed2cde0b287623624cd9bb75fd00ab8f7a9da4b7e07cc4ec5ac1d1d3e\": container with ID starting with 002618aed2cde0b287623624cd9bb75fd00ab8f7a9da4b7e07cc4ec5ac1d1d3e not found: ID does not exist" Oct 30 22:35:04.441881 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:04.441812 4335 scope.go:122] "RemoveContainer" containerID="f305f04e50ee079295e625005f01f05fef22490b4cc3c54a926f557a040df7b4" Oct 30 22:35:04.442075 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:35:04.442058 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f305f04e50ee079295e625005f01f05fef22490b4cc3c54a926f557a040df7b4\": container with ID starting with f305f04e50ee079295e625005f01f05fef22490b4cc3c54a926f557a040df7b4 not found: ID does not exist" containerID="f305f04e50ee079295e625005f01f05fef22490b4cc3c54a926f557a040df7b4" Oct 30 22:35:04.442115 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:04.442082 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f305f04e50ee079295e625005f01f05fef22490b4cc3c54a926f557a040df7b4"} err="failed to get container status \"f305f04e50ee079295e625005f01f05fef22490b4cc3c54a926f557a040df7b4\": rpc error: code = NotFound desc = could not find container \"f305f04e50ee079295e625005f01f05fef22490b4cc3c54a926f557a040df7b4\": container with ID starting with f305f04e50ee079295e625005f01f05fef22490b4cc3c54a926f557a040df7b4 not found: ID does not exist" Oct 30 22:35:04.585517 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:04.585488 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4"] Oct 30 22:35:04.634780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:04.634756 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-026c11-openshift-default-5884675f97-6szq4"] Oct 30 22:35:05.058386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:05.058356 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1657e3aa-0cfe-49e5-b8ce-21888a334461" path="/var/lib/kubelet/pods/1657e3aa-0cfe-49e5-b8ce-21888a334461/volumes" Oct 30 22:35:07.415675 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:07.415637 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7"] Oct 30 22:35:07.533242 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:07.533204 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt"] Oct 30 22:35:09.959840 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:09.959804 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk"] Oct 30 22:35:10.065414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:10.065378 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q"] Oct 30 22:35:12.416023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:12.415984 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7" podUID="32093b24-028d-4771-8aff-db94898573cd" containerName="payload-processing" containerID="cri-o://dc8158f4620dde8a7a3654839434903c49930f4db77bce7d269ac878b797d1b7" gracePeriod=25 Oct 30 22:35:12.533845 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:12.533817 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" podUID="138f5ad6-0075-4afe-abb6-526d5c89d7ad" containerName="payload-pre-processing" containerID="cri-o://aae2c44a71468de9a2d3c8652184bd1676888f0e26e2ef2a8ffa2896255d3749" gracePeriod=25 Oct 30 22:35:12.665312 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:12.665289 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7" Oct 30 22:35:12.809981 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:12.809951 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32093b24-028d-4771-8aff-db94898573cd-kube-api-access-d9knb\" (UniqueName: \"kubernetes.io/projected/32093b24-028d-4771-8aff-db94898573cd-kube-api-access-d9knb\") pod \"32093b24-028d-4771-8aff-db94898573cd\" (UID: \"32093b24-028d-4771-8aff-db94898573cd\") " Oct 30 22:35:12.810142 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:12.810034 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32093b24-028d-4771-8aff-db94898573cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32093b24-028d-4771-8aff-db94898573cd-plugins-config-volume\") pod \"32093b24-028d-4771-8aff-db94898573cd\" (UID: \"32093b24-028d-4771-8aff-db94898573cd\") " Oct 30 22:35:12.810403 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:12.810382 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32093b24-028d-4771-8aff-db94898573cd-plugins-config-volume" pod "32093b24-028d-4771-8aff-db94898573cd" (UID: "32093b24-028d-4771-8aff-db94898573cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:12.812112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:12.812091 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32093b24-028d-4771-8aff-db94898573cd-kube-api-access-d9knb" pod "32093b24-028d-4771-8aff-db94898573cd" (UID: "32093b24-028d-4771-8aff-db94898573cd"). InnerVolumeSpecName "kube-api-access-d9knb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:12.911443 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:12.911416 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9knb\" (UniqueName: \"kubernetes.io/projected/32093b24-028d-4771-8aff-db94898573cd-kube-api-access-d9knb\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:12.911443 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:12.911441 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32093b24-028d-4771-8aff-db94898573cd-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:13.481395 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:13.481312 4335 generic.go:356] "Generic (PLEG): container finished" podID="32093b24-028d-4771-8aff-db94898573cd" containerID="dc8158f4620dde8a7a3654839434903c49930f4db77bce7d269ac878b797d1b7" exitCode=0 Oct 30 22:35:13.481395 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:13.481357 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7" Oct 30 22:35:13.481901 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:13.481393 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7" event={"ID":"32093b24-028d-4771-8aff-db94898573cd","Type":"ContainerDied","Data":"dc8158f4620dde8a7a3654839434903c49930f4db77bce7d269ac878b797d1b7"} Oct 30 22:35:13.481901 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:13.481449 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7" event={"ID":"32093b24-028d-4771-8aff-db94898573cd","Type":"ContainerDied","Data":"d73cdd5616d24b44944c43bc8cef3e371e9a5ebd0c789eaa1c80b63bb7b0175b"} Oct 30 22:35:13.481901 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:13.481474 4335 scope.go:122] "RemoveContainer" containerID="dc8158f4620dde8a7a3654839434903c49930f4db77bce7d269ac878b797d1b7" Oct 30 22:35:13.491308 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:13.491275 4335 scope.go:122] "RemoveContainer" containerID="dc8158f4620dde8a7a3654839434903c49930f4db77bce7d269ac878b797d1b7" Oct 30 22:35:13.491578 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:35:13.491560 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc8158f4620dde8a7a3654839434903c49930f4db77bce7d269ac878b797d1b7\": container with ID starting with dc8158f4620dde8a7a3654839434903c49930f4db77bce7d269ac878b797d1b7 not found: ID does not exist" containerID="dc8158f4620dde8a7a3654839434903c49930f4db77bce7d269ac878b797d1b7" Oct 30 22:35:13.491666 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:13.491586 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc8158f4620dde8a7a3654839434903c49930f4db77bce7d269ac878b797d1b7"} err="failed to get container status \"dc8158f4620dde8a7a3654839434903c49930f4db77bce7d269ac878b797d1b7\": rpc error: code = NotFound desc = could not find container \"dc8158f4620dde8a7a3654839434903c49930f4db77bce7d269ac878b797d1b7\": container with ID starting with dc8158f4620dde8a7a3654839434903c49930f4db77bce7d269ac878b797d1b7 not found: ID does not exist" Oct 30 22:35:13.650669 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:13.650635 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7"] Oct 30 22:35:13.699760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:13.699739 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c04440-7889cc999c-7g9s7"] Oct 30 22:35:14.960600 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:14.960564 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" podUID="0c240af7-e207-49f2-b539-8136924ff809" containerName="payload-processing" containerID="cri-o://98bcf501b1440d508565bcf37113b1a0d1d6c0d1e00aee8d65fd045623b93a47" gracePeriod=25 Oct 30 22:35:15.057842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:15.057814 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32093b24-028d-4771-8aff-db94898573cd" path="/var/lib/kubelet/pods/32093b24-028d-4771-8aff-db94898573cd/volumes" Oct 30 22:35:15.066651 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:15.066622 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" podUID="96dea67c-e674-4c4e-81d3-0c5bea462f7c" containerName="payload-pre-processing" containerID="cri-o://e75178bdcc15529ebecefd7a726478ee5f7a856d6aa12d8797950fe371fcf045" gracePeriod=25 Oct 30 22:35:15.445268 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:15.445229 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" podUID="138f5ad6-0075-4afe-abb6-526d5c89d7ad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:35:21.310439 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:21.310407 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" podUID="96dea67c-e674-4c4e-81d3-0c5bea462f7c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:35:21.310439 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:21.310431 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" podUID="0c240af7-e207-49f2-b539-8136924ff809" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:35:25.322188 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:35:25.322144 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:35:25.445268 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:25.445231 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" podUID="138f5ad6-0075-4afe-abb6-526d5c89d7ad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:35:27.152338 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:27.152309 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:35:28.411969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:28.411922 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn"] Oct 30 22:35:28.460169 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:28.460140 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32dd8d-zgwvn"] Oct 30 22:35:29.058854 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:29.058818 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="205109aa-7cf4-433c-8657-6c3174229ef4" path="/var/lib/kubelet/pods/205109aa-7cf4-433c-8657-6c3174229ef4/volumes" Oct 30 22:35:31.072472 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.072439 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7"] Oct 30 22:35:31.072949 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.072932 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f2ad9d54-24dc-40ea-9397-5bfc51b9d786" containerName="istio-proxy" Oct 30 22:35:31.072995 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.072950 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ad9d54-24dc-40ea-9397-5bfc51b9d786" containerName="istio-proxy" Oct 30 22:35:31.072995 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.072976 4335 cpu_manager.go:414] "containerMap: removing container" podUID="205109aa-7cf4-433c-8657-6c3174229ef4" containerName="revoke-keys" Oct 30 22:35:31.072995 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.072982 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="205109aa-7cf4-433c-8657-6c3174229ef4" containerName="revoke-keys" Oct 30 22:35:31.072995 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.072992 4335 cpu_manager.go:414] "containerMap: removing container" podUID="1657e3aa-0cfe-49e5-b8ce-21888a334461" containerName="istio-proxy" Oct 30 22:35:31.073129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.072997 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1657e3aa-0cfe-49e5-b8ce-21888a334461" containerName="istio-proxy" Oct 30 22:35:31.073129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.073007 4335 cpu_manager.go:414] "containerMap: removing container" podUID="1657e3aa-0cfe-49e5-b8ce-21888a334461" containerName="istio-proxy" Oct 30 22:35:31.073129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.073012 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1657e3aa-0cfe-49e5-b8ce-21888a334461" containerName="istio-proxy" Oct 30 22:35:31.073129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.073022 4335 cpu_manager.go:414] "containerMap: removing container" podUID="32093b24-028d-4771-8aff-db94898573cd" containerName="payload-processing" Oct 30 22:35:31.073129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.073028 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32093b24-028d-4771-8aff-db94898573cd" containerName="payload-processing" Oct 30 22:35:31.073129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.073096 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ad9d54-24dc-40ea-9397-5bfc51b9d786" containerName="istio-proxy" Oct 30 22:35:31.073129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.073103 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="205109aa-7cf4-433c-8657-6c3174229ef4" containerName="revoke-keys" Oct 30 22:35:31.073129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.073110 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="1657e3aa-0cfe-49e5-b8ce-21888a334461" containerName="istio-proxy" Oct 30 22:35:31.073129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.073115 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="1657e3aa-0cfe-49e5-b8ce-21888a334461" containerName="istio-proxy" Oct 30 22:35:31.073129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.073123 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="32093b24-028d-4771-8aff-db94898573cd" containerName="payload-processing" Oct 30 22:35:31.077937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.077917 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7" Oct 30 22:35:31.135129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.135106 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7"] Oct 30 22:35:31.164420 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.164392 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7\" (UID: \"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7" Oct 30 22:35:31.164542 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.164515 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7\" (UID: \"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7" Oct 30 22:35:31.164595 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.164544 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwkp7\" (UniqueName: \"kubernetes.io/projected/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-kube-api-access-rwkp7\") pod \"maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7\" (UID: \"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7" Oct 30 22:35:31.180836 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.180814 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:35:31.180836 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.180815 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:35:31.181005 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.180815 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:35:31.181005 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.180865 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:35:31.265726 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.265701 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7\" (UID: \"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7" Oct 30 22:35:31.265851 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.265731 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwkp7\" (UniqueName: \"kubernetes.io/projected/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-kube-api-access-rwkp7\") pod \"maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7\" (UID: \"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7" Oct 30 22:35:31.265851 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.265810 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7\" (UID: \"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7" Oct 30 22:35:31.266419 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.266396 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7\" (UID: \"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7" Oct 30 22:35:31.268217 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.268193 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7\" (UID: \"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7" Oct 30 22:35:31.310212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.310187 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" podUID="96dea67c-e674-4c4e-81d3-0c5bea462f7c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:35:31.310337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.310218 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" podUID="0c240af7-e207-49f2-b539-8136924ff809" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:35:31.330571 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.330520 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwkp7\" (UniqueName: \"kubernetes.io/projected/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-kube-api-access-rwkp7\") pod \"maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7\" (UID: \"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7" Oct 30 22:35:31.391217 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.391198 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7" Oct 30 22:35:31.776708 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:31.776683 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7"] Oct 30 22:35:31.778171 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:35:31.778141 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbae0c7_0910_40f2_bd8c_ef93df5ddee8.slice/crio-4d19ea8071b0c993a7716849800e4f88e51ce0c5bc97419d5969dffe8366944b WatchSource:0}: Error finding container 4d19ea8071b0c993a7716849800e4f88e51ce0c5bc97419d5969dffe8366944b: Status 404 returned error can't find the container with id 4d19ea8071b0c993a7716849800e4f88e51ce0c5bc97419d5969dffe8366944b Oct 30 22:35:32.605790 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:32.605761 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7" event={"ID":"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8","Type":"ContainerStarted","Data":"e4fe8d7212e4f0767c5b10d6b684f52b6cc6fc1d698a40a8f68f67413ed16212"} Oct 30 22:35:32.606199 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:32.605817 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7" event={"ID":"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8","Type":"ContainerStarted","Data":"4d19ea8071b0c993a7716849800e4f88e51ce0c5bc97419d5969dffe8366944b"} Oct 30 22:35:32.721934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:32.721874 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7" podStartSLOduration=1.7218555169999998 podStartE2EDuration="1.721855517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:32.720733663 +0000 UTC m=+3065.821165398" watchObservedRunningTime="2026-10-30 22:35:32.721855517 +0000 UTC m=+3065.822287249" Oct 30 22:35:32.785444 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:32.785417 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" Oct 30 22:35:32.883517 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:32.883486 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/138f5ad6-0075-4afe-abb6-526d5c89d7ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/138f5ad6-0075-4afe-abb6-526d5c89d7ad-plugins-config-volume\") pod \"138f5ad6-0075-4afe-abb6-526d5c89d7ad\" (UID: \"138f5ad6-0075-4afe-abb6-526d5c89d7ad\") " Oct 30 22:35:32.883692 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:32.883550 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138f5ad6-0075-4afe-abb6-526d5c89d7ad-kube-api-access-57d7d\" (UniqueName: \"kubernetes.io/projected/138f5ad6-0075-4afe-abb6-526d5c89d7ad-kube-api-access-57d7d\") pod \"138f5ad6-0075-4afe-abb6-526d5c89d7ad\" (UID: \"138f5ad6-0075-4afe-abb6-526d5c89d7ad\") " Oct 30 22:35:32.883872 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:32.883852 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/138f5ad6-0075-4afe-abb6-526d5c89d7ad-plugins-config-volume" pod "138f5ad6-0075-4afe-abb6-526d5c89d7ad" (UID: "138f5ad6-0075-4afe-abb6-526d5c89d7ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:32.885656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:32.885637 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138f5ad6-0075-4afe-abb6-526d5c89d7ad-kube-api-access-57d7d" pod "138f5ad6-0075-4afe-abb6-526d5c89d7ad" (UID: "138f5ad6-0075-4afe-abb6-526d5c89d7ad"). InnerVolumeSpecName "kube-api-access-57d7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:32.984404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:32.984374 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/138f5ad6-0075-4afe-abb6-526d5c89d7ad-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:32.984404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:32.984401 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57d7d\" (UniqueName: \"kubernetes.io/projected/138f5ad6-0075-4afe-abb6-526d5c89d7ad-kube-api-access-57d7d\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:33.091714 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:35:33.091689 4335 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138f5ad6_0075_4afe_abb6_526d5c89d7ad.slice/crio-3b75afc0efa9b16ee131d643afe4e7ab57e6070a8fe71e06adba50cb889d7cf4\": RecentStats: unable to find data in memory cache]" Oct 30 22:35:33.613257 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:33.613222 4335 generic.go:356] "Generic (PLEG): container finished" podID="ddbae0c7-0910-40f2-bd8c-ef93df5ddee8" containerID="e4fe8d7212e4f0767c5b10d6b684f52b6cc6fc1d698a40a8f68f67413ed16212" exitCode=0 Oct 30 22:35:33.613724 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:33.613304 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7" event={"ID":"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8","Type":"ContainerDied","Data":"e4fe8d7212e4f0767c5b10d6b684f52b6cc6fc1d698a40a8f68f67413ed16212"} Oct 30 22:35:33.614892 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:33.614861 4335 generic.go:356] "Generic (PLEG): container finished" podID="138f5ad6-0075-4afe-abb6-526d5c89d7ad" containerID="aae2c44a71468de9a2d3c8652184bd1676888f0e26e2ef2a8ffa2896255d3749" exitCode=0 Oct 30 22:35:33.615033 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:33.614899 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" event={"ID":"138f5ad6-0075-4afe-abb6-526d5c89d7ad","Type":"ContainerDied","Data":"aae2c44a71468de9a2d3c8652184bd1676888f0e26e2ef2a8ffa2896255d3749"} Oct 30 22:35:33.615033 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:33.614943 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" event={"ID":"138f5ad6-0075-4afe-abb6-526d5c89d7ad","Type":"ContainerDied","Data":"3b75afc0efa9b16ee131d643afe4e7ab57e6070a8fe71e06adba50cb889d7cf4"} Oct 30 22:35:33.615033 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:33.614964 4335 scope.go:122] "RemoveContainer" containerID="aae2c44a71468de9a2d3c8652184bd1676888f0e26e2ef2a8ffa2896255d3749" Oct 30 22:35:33.615033 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:33.614906 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt" Oct 30 22:35:33.628295 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:33.628278 4335 scope.go:122] "RemoveContainer" containerID="aae2c44a71468de9a2d3c8652184bd1676888f0e26e2ef2a8ffa2896255d3749" Oct 30 22:35:33.628587 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:35:33.628564 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae2c44a71468de9a2d3c8652184bd1676888f0e26e2ef2a8ffa2896255d3749\": container with ID starting with aae2c44a71468de9a2d3c8652184bd1676888f0e26e2ef2a8ffa2896255d3749 not found: ID does not exist" containerID="aae2c44a71468de9a2d3c8652184bd1676888f0e26e2ef2a8ffa2896255d3749" Oct 30 22:35:33.628671 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:33.628595 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae2c44a71468de9a2d3c8652184bd1676888f0e26e2ef2a8ffa2896255d3749"} err="failed to get container status \"aae2c44a71468de9a2d3c8652184bd1676888f0e26e2ef2a8ffa2896255d3749\": rpc error: code = NotFound desc = could not find container \"aae2c44a71468de9a2d3c8652184bd1676888f0e26e2ef2a8ffa2896255d3749\": container with ID starting with aae2c44a71468de9a2d3c8652184bd1676888f0e26e2ef2a8ffa2896255d3749 not found: ID does not exist" Oct 30 22:35:33.850106 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:33.850071 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt"] Oct 30 22:35:33.855656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:33.855632 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c04440-7675d76f99-pkjlt"] Oct 30 22:35:34.775795 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:34.775772 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7" Oct 30 22:35:34.802110 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:34.802085 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-maas-api-xks-ca\") pod \"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8\" (UID: \"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8\") " Oct 30 22:35:34.802229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:34.802160 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-maas-api-openshift-ca\") pod \"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8\" (UID: \"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8\") " Oct 30 22:35:34.802287 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:34.802274 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-kube-api-access-rwkp7\" (UniqueName: \"kubernetes.io/projected/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-kube-api-access-rwkp7\") pod \"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8\" (UID: \"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8\") " Oct 30 22:35:34.802535 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:34.802507 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-maas-api-openshift-ca" pod "ddbae0c7-0910-40f2-bd8c-ef93df5ddee8" (UID: "ddbae0c7-0910-40f2-bd8c-ef93df5ddee8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:34.802741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:34.802702 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:34.804578 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:34.804554 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-kube-api-access-rwkp7" pod "ddbae0c7-0910-40f2-bd8c-ef93df5ddee8" (UID: "ddbae0c7-0910-40f2-bd8c-ef93df5ddee8"). InnerVolumeSpecName "kube-api-access-rwkp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:34.804789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:34.804767 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-maas-api-xks-ca" pod "ddbae0c7-0910-40f2-bd8c-ef93df5ddee8" (UID: "ddbae0c7-0910-40f2-bd8c-ef93df5ddee8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:34.904350 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:34.904282 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwkp7\" (UniqueName: \"kubernetes.io/projected/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-kube-api-access-rwkp7\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:34.904350 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:34.904318 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:35.056888 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.056857 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="138f5ad6-0075-4afe-abb6-526d5c89d7ad" path="/var/lib/kubelet/pods/138f5ad6-0075-4afe-abb6-526d5c89d7ad/volumes" Oct 30 22:35:35.208593 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.208572 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" Oct 30 22:35:35.308689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.308663 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c240af7-e207-49f2-b539-8136924ff809-kube-api-access-pp2fj\" (UniqueName: \"kubernetes.io/projected/0c240af7-e207-49f2-b539-8136924ff809-kube-api-access-pp2fj\") pod \"0c240af7-e207-49f2-b539-8136924ff809\" (UID: \"0c240af7-e207-49f2-b539-8136924ff809\") " Oct 30 22:35:35.308825 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.308737 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c240af7-e207-49f2-b539-8136924ff809-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c240af7-e207-49f2-b539-8136924ff809-plugins-config-volume\") pod \"0c240af7-e207-49f2-b539-8136924ff809\" (UID: \"0c240af7-e207-49f2-b539-8136924ff809\") " Oct 30 22:35:35.309188 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.309163 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c240af7-e207-49f2-b539-8136924ff809-plugins-config-volume" pod "0c240af7-e207-49f2-b539-8136924ff809" (UID: "0c240af7-e207-49f2-b539-8136924ff809"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:35.311017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.310993 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c240af7-e207-49f2-b539-8136924ff809-kube-api-access-pp2fj" pod "0c240af7-e207-49f2-b539-8136924ff809" (UID: "0c240af7-e207-49f2-b539-8136924ff809"). InnerVolumeSpecName "kube-api-access-pp2fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:35.316634 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.316601 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" Oct 30 22:35:35.409973 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.409920 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96dea67c-e674-4c4e-81d3-0c5bea462f7c-kube-api-access-g7gc6\" (UniqueName: \"kubernetes.io/projected/96dea67c-e674-4c4e-81d3-0c5bea462f7c-kube-api-access-g7gc6\") pod \"96dea67c-e674-4c4e-81d3-0c5bea462f7c\" (UID: \"96dea67c-e674-4c4e-81d3-0c5bea462f7c\") " Oct 30 22:35:35.410137 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.409986 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96dea67c-e674-4c4e-81d3-0c5bea462f7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96dea67c-e674-4c4e-81d3-0c5bea462f7c-plugins-config-volume\") pod \"96dea67c-e674-4c4e-81d3-0c5bea462f7c\" (UID: \"96dea67c-e674-4c4e-81d3-0c5bea462f7c\") " Oct 30 22:35:35.410378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.410352 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp2fj\" (UniqueName: \"kubernetes.io/projected/0c240af7-e207-49f2-b539-8136924ff809-kube-api-access-pp2fj\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:35.410484 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.410383 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c240af7-e207-49f2-b539-8136924ff809-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:35.410786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.410758 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96dea67c-e674-4c4e-81d3-0c5bea462f7c-plugins-config-volume" pod "96dea67c-e674-4c4e-81d3-0c5bea462f7c" (UID: "96dea67c-e674-4c4e-81d3-0c5bea462f7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:35.413808 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.413783 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96dea67c-e674-4c4e-81d3-0c5bea462f7c-kube-api-access-g7gc6" pod "96dea67c-e674-4c4e-81d3-0c5bea462f7c" (UID: "96dea67c-e674-4c4e-81d3-0c5bea462f7c"). InnerVolumeSpecName "kube-api-access-g7gc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:35.437760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.437681 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5"] Oct 30 22:35:35.437905 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.437886 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" podUID="c280505e-e32e-4a17-9643-3be7e1250e0d" containerName="istio-proxy" containerID="cri-o://427e922cc7671ba9a8af3d009d4cb7fc8bc4c650a187554c0c9258a7373d07ba" gracePeriod=30 Oct 30 22:35:35.511395 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.511370 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7gc6\" (UniqueName: \"kubernetes.io/projected/96dea67c-e674-4c4e-81d3-0c5bea462f7c-kube-api-access-g7gc6\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:35.511395 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.511393 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96dea67c-e674-4c4e-81d3-0c5bea462f7c-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:35.633010 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.632974 4335 generic.go:356] "Generic (PLEG): container finished" podID="0c240af7-e207-49f2-b539-8136924ff809" containerID="98bcf501b1440d508565bcf37113b1a0d1d6c0d1e00aee8d65fd045623b93a47" exitCode=0 Oct 30 22:35:35.633206 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.633026 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" Oct 30 22:35:35.633206 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.633067 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" event={"ID":"0c240af7-e207-49f2-b539-8136924ff809","Type":"ContainerDied","Data":"98bcf501b1440d508565bcf37113b1a0d1d6c0d1e00aee8d65fd045623b93a47"} Oct 30 22:35:35.633206 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.633123 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk" event={"ID":"0c240af7-e207-49f2-b539-8136924ff809","Type":"ContainerDied","Data":"e150ac3a9b527a9bbe74d0abb7452c33fdd2a2900159954cdc0012e9d8c7d818"} Oct 30 22:35:35.633206 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.633150 4335 scope.go:122] "RemoveContainer" containerID="98bcf501b1440d508565bcf37113b1a0d1d6c0d1e00aee8d65fd045623b93a47" Oct 30 22:35:35.634599 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.634582 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7" Oct 30 22:35:35.634731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.634578 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7" event={"ID":"ddbae0c7-0910-40f2-bd8c-ef93df5ddee8","Type":"ContainerDied","Data":"4d19ea8071b0c993a7716849800e4f88e51ce0c5bc97419d5969dffe8366944b"} Oct 30 22:35:35.634731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.634655 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d19ea8071b0c993a7716849800e4f88e51ce0c5bc97419d5969dffe8366944b" Oct 30 22:35:35.636881 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.636855 4335 generic.go:356] "Generic (PLEG): container finished" podID="96dea67c-e674-4c4e-81d3-0c5bea462f7c" containerID="e75178bdcc15529ebecefd7a726478ee5f7a856d6aa12d8797950fe371fcf045" exitCode=0 Oct 30 22:35:35.637023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.636909 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" Oct 30 22:35:35.637023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.636962 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" event={"ID":"96dea67c-e674-4c4e-81d3-0c5bea462f7c","Type":"ContainerDied","Data":"e75178bdcc15529ebecefd7a726478ee5f7a856d6aa12d8797950fe371fcf045"} Oct 30 22:35:35.637023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.637001 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q" event={"ID":"96dea67c-e674-4c4e-81d3-0c5bea462f7c","Type":"ContainerDied","Data":"c0ba0b1fe24438903dbd2198e0132d67d292fccb94b87152c14a859113dbc6c2"} Oct 30 22:35:35.653429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.653413 4335 scope.go:122] "RemoveContainer" containerID="98bcf501b1440d508565bcf37113b1a0d1d6c0d1e00aee8d65fd045623b93a47" Oct 30 22:35:35.653729 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:35:35.653710 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98bcf501b1440d508565bcf37113b1a0d1d6c0d1e00aee8d65fd045623b93a47\": container with ID starting with 98bcf501b1440d508565bcf37113b1a0d1d6c0d1e00aee8d65fd045623b93a47 not found: ID does not exist" containerID="98bcf501b1440d508565bcf37113b1a0d1d6c0d1e00aee8d65fd045623b93a47" Oct 30 22:35:35.653780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.653736 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98bcf501b1440d508565bcf37113b1a0d1d6c0d1e00aee8d65fd045623b93a47"} err="failed to get container status \"98bcf501b1440d508565bcf37113b1a0d1d6c0d1e00aee8d65fd045623b93a47\": rpc error: code = NotFound desc = could not find container \"98bcf501b1440d508565bcf37113b1a0d1d6c0d1e00aee8d65fd045623b93a47\": container with ID starting with 98bcf501b1440d508565bcf37113b1a0d1d6c0d1e00aee8d65fd045623b93a47 not found: ID does not exist" Oct 30 22:35:35.653780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.653754 4335 scope.go:122] "RemoveContainer" containerID="e75178bdcc15529ebecefd7a726478ee5f7a856d6aa12d8797950fe371fcf045" Oct 30 22:35:35.659734 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.659716 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68"] Oct 30 22:35:35.663408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.663389 4335 scope.go:122] "RemoveContainer" containerID="e75178bdcc15529ebecefd7a726478ee5f7a856d6aa12d8797950fe371fcf045" Oct 30 22:35:35.663722 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:35:35.663697 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e75178bdcc15529ebecefd7a726478ee5f7a856d6aa12d8797950fe371fcf045\": container with ID starting with e75178bdcc15529ebecefd7a726478ee5f7a856d6aa12d8797950fe371fcf045 not found: ID does not exist" containerID="e75178bdcc15529ebecefd7a726478ee5f7a856d6aa12d8797950fe371fcf045" Oct 30 22:35:35.663810 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.663730 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e75178bdcc15529ebecefd7a726478ee5f7a856d6aa12d8797950fe371fcf045"} err="failed to get container status \"e75178bdcc15529ebecefd7a726478ee5f7a856d6aa12d8797950fe371fcf045\": rpc error: code = NotFound desc = could not find container \"e75178bdcc15529ebecefd7a726478ee5f7a856d6aa12d8797950fe371fcf045\": container with ID starting with e75178bdcc15529ebecefd7a726478ee5f7a856d6aa12d8797950fe371fcf045 not found: ID does not exist" Oct 30 22:35:35.869200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.869173 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q"] Oct 30 22:35:35.872799 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:35.872777 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-32dd8d-847854b8c-qw28q"] Oct 30 22:35:36.095445 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.095420 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk"] Oct 30 22:35:36.100386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.100363 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-32dd8d-5d4fd65cf7-w98rk"] Oct 30 22:35:36.417266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.417190 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q"] Oct 30 22:35:36.417682 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.417667 4335 cpu_manager.go:414] "containerMap: removing container" podUID="ddbae0c7-0910-40f2-bd8c-ef93df5ddee8" containerName="revoke-keys" Oct 30 22:35:36.417741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.417683 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddbae0c7-0910-40f2-bd8c-ef93df5ddee8" containerName="revoke-keys" Oct 30 22:35:36.417741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.417696 4335 cpu_manager.go:414] "containerMap: removing container" podUID="0c240af7-e207-49f2-b539-8136924ff809" containerName="payload-processing" Oct 30 22:35:36.417741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.417704 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c240af7-e207-49f2-b539-8136924ff809" containerName="payload-processing" Oct 30 22:35:36.417741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.417715 4335 cpu_manager.go:414] "containerMap: removing container" podUID="138f5ad6-0075-4afe-abb6-526d5c89d7ad" containerName="payload-pre-processing" Oct 30 22:35:36.417741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.417721 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="138f5ad6-0075-4afe-abb6-526d5c89d7ad" containerName="payload-pre-processing" Oct 30 22:35:36.417741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.417735 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f2ad9d54-24dc-40ea-9397-5bfc51b9d786" containerName="istio-proxy" Oct 30 22:35:36.417741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.417741 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ad9d54-24dc-40ea-9397-5bfc51b9d786" containerName="istio-proxy" Oct 30 22:35:36.417962 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.417759 4335 cpu_manager.go:414] "containerMap: removing container" podUID="96dea67c-e674-4c4e-81d3-0c5bea462f7c" containerName="payload-pre-processing" Oct 30 22:35:36.417962 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.417764 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96dea67c-e674-4c4e-81d3-0c5bea462f7c" containerName="payload-pre-processing" Oct 30 22:35:36.417962 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.417839 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="96dea67c-e674-4c4e-81d3-0c5bea462f7c" containerName="payload-pre-processing" Oct 30 22:35:36.417962 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.417850 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ad9d54-24dc-40ea-9397-5bfc51b9d786" containerName="istio-proxy" Oct 30 22:35:36.417962 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.417860 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddbae0c7-0910-40f2-bd8c-ef93df5ddee8" containerName="revoke-keys" Oct 30 22:35:36.417962 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.417870 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c240af7-e207-49f2-b539-8136924ff809" containerName="payload-processing" Oct 30 22:35:36.417962 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.417876 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="138f5ad6-0075-4afe-abb6-526d5c89d7ad" containerName="payload-pre-processing" Oct 30 22:35:36.422415 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.422392 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q" Oct 30 22:35:36.474981 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.474959 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:35:36.475149 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.475039 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:35:36.478515 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.478492 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q"] Oct 30 22:35:36.501040 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.501002 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:35:36.501251 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.501234 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:35:36.522814 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.522783 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57af8388-c78d-4617-b19a-2adfc7d14e11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q\" (UID: \"57af8388-c78d-4617-b19a-2adfc7d14e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q" Oct 30 22:35:36.522960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.522864 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57af8388-c78d-4617-b19a-2adfc7d14e11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q\" (UID: \"57af8388-c78d-4617-b19a-2adfc7d14e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q" Oct 30 22:35:36.523065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.523039 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z694\" (UniqueName: \"kubernetes.io/projected/57af8388-c78d-4617-b19a-2adfc7d14e11-kube-api-access-5z694\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q\" (UID: \"57af8388-c78d-4617-b19a-2adfc7d14e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q" Oct 30 22:35:36.623919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.623884 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57af8388-c78d-4617-b19a-2adfc7d14e11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q\" (UID: \"57af8388-c78d-4617-b19a-2adfc7d14e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q" Oct 30 22:35:36.624069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.623975 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z694\" (UniqueName: \"kubernetes.io/projected/57af8388-c78d-4617-b19a-2adfc7d14e11-kube-api-access-5z694\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q\" (UID: \"57af8388-c78d-4617-b19a-2adfc7d14e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q" Oct 30 22:35:36.624069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.624011 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57af8388-c78d-4617-b19a-2adfc7d14e11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q\" (UID: \"57af8388-c78d-4617-b19a-2adfc7d14e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q" Oct 30 22:35:36.624543 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.624522 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57af8388-c78d-4617-b19a-2adfc7d14e11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q\" (UID: \"57af8388-c78d-4617-b19a-2adfc7d14e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q" Oct 30 22:35:36.626720 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.626691 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57af8388-c78d-4617-b19a-2adfc7d14e11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q\" (UID: \"57af8388-c78d-4617-b19a-2adfc7d14e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q" Oct 30 22:35:36.684385 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.684326 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z694\" (UniqueName: \"kubernetes.io/projected/57af8388-c78d-4617-b19a-2adfc7d14e11-kube-api-access-5z694\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q\" (UID: \"57af8388-c78d-4617-b19a-2adfc7d14e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q" Oct 30 22:35:36.736012 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:36.735991 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q" Oct 30 22:35:37.057654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:37.057603 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c240af7-e207-49f2-b539-8136924ff809" path="/var/lib/kubelet/pods/0c240af7-e207-49f2-b539-8136924ff809/volumes" Oct 30 22:35:37.058119 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:37.057991 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96dea67c-e674-4c4e-81d3-0c5bea462f7c" path="/var/lib/kubelet/pods/96dea67c-e674-4c4e-81d3-0c5bea462f7c/volumes" Oct 30 22:35:37.116020 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:37.115987 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q"] Oct 30 22:35:37.118328 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:35:37.118302 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57af8388_c78d_4617_b19a_2adfc7d14e11.slice/crio-24501774f5e8a1fd3ca4aba3b6d31e0f181273fe7e40b7f574bfee847a1fddf6 WatchSource:0}: Error finding container 24501774f5e8a1fd3ca4aba3b6d31e0f181273fe7e40b7f574bfee847a1fddf6: Status 404 returned error can't find the container with id 24501774f5e8a1fd3ca4aba3b6d31e0f181273fe7e40b7f574bfee847a1fddf6 Oct 30 22:35:37.654502 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:37.654436 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q" event={"ID":"57af8388-c78d-4617-b19a-2adfc7d14e11","Type":"ContainerStarted","Data":"2d82618e492818a407b7cc67e939a710edd594ef570e07b681f1bdc88379775d"} Oct 30 22:35:37.654674 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:37.654519 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q" event={"ID":"57af8388-c78d-4617-b19a-2adfc7d14e11","Type":"ContainerStarted","Data":"24501774f5e8a1fd3ca4aba3b6d31e0f181273fe7e40b7f574bfee847a1fddf6"} Oct 30 22:35:37.778592 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:37.778546 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q" podStartSLOduration=1.7785319450000001 podStartE2EDuration="1.778531945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:37.777174547 +0000 UTC m=+3070.877606292" watchObservedRunningTime="2026-10-30 22:35:37.778531945 +0000 UTC m=+3070.878963702" Oct 30 22:35:38.662181 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:38.662146 4335 generic.go:356] "Generic (PLEG): container finished" podID="57af8388-c78d-4617-b19a-2adfc7d14e11" containerID="2d82618e492818a407b7cc67e939a710edd594ef570e07b681f1bdc88379775d" exitCode=0 Oct 30 22:35:38.662584 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:38.662224 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q" event={"ID":"57af8388-c78d-4617-b19a-2adfc7d14e11","Type":"ContainerDied","Data":"2d82618e492818a407b7cc67e939a710edd594ef570e07b681f1bdc88379775d"} Oct 30 22:35:39.814595 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:39.814569 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q" Oct 30 22:35:39.857985 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:39.857953 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57af8388-c78d-4617-b19a-2adfc7d14e11-kube-api-access-5z694\" (UniqueName: \"kubernetes.io/projected/57af8388-c78d-4617-b19a-2adfc7d14e11-kube-api-access-5z694\") pod \"57af8388-c78d-4617-b19a-2adfc7d14e11\" (UID: \"57af8388-c78d-4617-b19a-2adfc7d14e11\") " Oct 30 22:35:39.858143 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:39.857996 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57af8388-c78d-4617-b19a-2adfc7d14e11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57af8388-c78d-4617-b19a-2adfc7d14e11-maas-api-openshift-ca\") pod \"57af8388-c78d-4617-b19a-2adfc7d14e11\" (UID: \"57af8388-c78d-4617-b19a-2adfc7d14e11\") " Oct 30 22:35:39.858205 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:39.858181 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57af8388-c78d-4617-b19a-2adfc7d14e11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57af8388-c78d-4617-b19a-2adfc7d14e11-maas-api-xks-ca\") pod \"57af8388-c78d-4617-b19a-2adfc7d14e11\" (UID: \"57af8388-c78d-4617-b19a-2adfc7d14e11\") " Oct 30 22:35:39.858370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:39.858347 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57af8388-c78d-4617-b19a-2adfc7d14e11-maas-api-openshift-ca" pod "57af8388-c78d-4617-b19a-2adfc7d14e11" (UID: "57af8388-c78d-4617-b19a-2adfc7d14e11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:39.858546 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:39.858531 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57af8388-c78d-4617-b19a-2adfc7d14e11-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:39.860262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:39.860242 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57af8388-c78d-4617-b19a-2adfc7d14e11-maas-api-xks-ca" pod "57af8388-c78d-4617-b19a-2adfc7d14e11" (UID: "57af8388-c78d-4617-b19a-2adfc7d14e11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:39.860361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:39.860268 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57af8388-c78d-4617-b19a-2adfc7d14e11-kube-api-access-5z694" pod "57af8388-c78d-4617-b19a-2adfc7d14e11" (UID: "57af8388-c78d-4617-b19a-2adfc7d14e11"). InnerVolumeSpecName "kube-api-access-5z694". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:39.959446 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:39.959375 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57af8388-c78d-4617-b19a-2adfc7d14e11-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:39.959446 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:39.959405 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z694\" (UniqueName: \"kubernetes.io/projected/57af8388-c78d-4617-b19a-2adfc7d14e11-kube-api-access-5z694\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:40.676800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.676772 4335 generic.go:356] "Generic (PLEG): container finished" podID="c280505e-e32e-4a17-9643-3be7e1250e0d" containerID="427e922cc7671ba9a8af3d009d4cb7fc8bc4c650a187554c0c9258a7373d07ba" exitCode=0 Oct 30 22:35:40.676934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.676840 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" event={"ID":"c280505e-e32e-4a17-9643-3be7e1250e0d","Type":"ContainerDied","Data":"427e922cc7671ba9a8af3d009d4cb7fc8bc4c650a187554c0c9258a7373d07ba"} Oct 30 22:35:40.676934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.676900 4335 scope.go:122] "RemoveContainer" containerID="7a4207df4bfb0e9afb11dcf4ff44a831c5a9b1572bc0515f7380fbab7d61610a" Oct 30 22:35:40.678687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.678657 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q" event={"ID":"57af8388-c78d-4617-b19a-2adfc7d14e11","Type":"ContainerDied","Data":"24501774f5e8a1fd3ca4aba3b6d31e0f181273fe7e40b7f574bfee847a1fddf6"} Oct 30 22:35:40.678687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.678684 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q" Oct 30 22:35:40.678851 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.678705 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24501774f5e8a1fd3ca4aba3b6d31e0f181273fe7e40b7f574bfee847a1fddf6" Oct 30 22:35:40.683333 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.683314 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:35:40.768149 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.768092 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c280505e-e32e-4a17-9643-3be7e1250e0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c280505e-e32e-4a17-9643-3be7e1250e0d-istiod-ca-cert\") pod \"c280505e-e32e-4a17-9643-3be7e1250e0d\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " Oct 30 22:35:40.768266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.768145 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-data\") pod \"c280505e-e32e-4a17-9643-3be7e1250e0d\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " Oct 30 22:35:40.768266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.768176 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-workload-socket\") pod \"c280505e-e32e-4a17-9643-3be7e1250e0d\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " Oct 30 22:35:40.768266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.768258 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-envoy\") pod \"c280505e-e32e-4a17-9643-3be7e1250e0d\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " Oct 30 22:35:40.768452 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.768308 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-podinfo\") pod \"c280505e-e32e-4a17-9643-3be7e1250e0d\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " Oct 30 22:35:40.768452 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.768363 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-token\" (UniqueName: \"kubernetes.io/projected/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-token\") pod \"c280505e-e32e-4a17-9643-3be7e1250e0d\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " Oct 30 22:35:40.768452 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.768417 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-workload-certs\") pod \"c280505e-e32e-4a17-9643-3be7e1250e0d\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " Oct 30 22:35:40.768452 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.768425 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c280505e-e32e-4a17-9643-3be7e1250e0d-istiod-ca-cert" pod "c280505e-e32e-4a17-9643-3be7e1250e0d" (UID: "c280505e-e32e-4a17-9643-3be7e1250e0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:40.768706 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.768453 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c280505e-e32e-4a17-9643-3be7e1250e0d-kube-api-access-jhgmc\" (UniqueName: \"kubernetes.io/projected/c280505e-e32e-4a17-9643-3be7e1250e0d-kube-api-access-jhgmc\") pod \"c280505e-e32e-4a17-9643-3be7e1250e0d\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " Oct 30 22:35:40.768706 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.768490 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-credential-socket\") pod \"c280505e-e32e-4a17-9643-3be7e1250e0d\" (UID: \"c280505e-e32e-4a17-9643-3be7e1250e0d\") " Oct 30 22:35:40.768819 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.768452 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-data" pod "c280505e-e32e-4a17-9643-3be7e1250e0d" (UID: "c280505e-e32e-4a17-9643-3be7e1250e0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:40.768938 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.768918 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c280505e-e32e-4a17-9643-3be7e1250e0d-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:40.769107 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.769091 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:40.769199 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.768944 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-credential-socket" pod "c280505e-e32e-4a17-9643-3be7e1250e0d" (UID: "c280505e-e32e-4a17-9643-3be7e1250e0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:40.769278 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.768547 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-workload-socket" pod "c280505e-e32e-4a17-9643-3be7e1250e0d" (UID: "c280505e-e32e-4a17-9643-3be7e1250e0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:40.769370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.768718 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-workload-certs" pod "c280505e-e32e-4a17-9643-3be7e1250e0d" (UID: "c280505e-e32e-4a17-9643-3be7e1250e0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:40.770826 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.770801 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c280505e-e32e-4a17-9643-3be7e1250e0d-kube-api-access-jhgmc" pod "c280505e-e32e-4a17-9643-3be7e1250e0d" (UID: "c280505e-e32e-4a17-9643-3be7e1250e0d"). InnerVolumeSpecName "kube-api-access-jhgmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:40.771160 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.771140 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-envoy" pod "c280505e-e32e-4a17-9643-3be7e1250e0d" (UID: "c280505e-e32e-4a17-9643-3be7e1250e0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:40.771160 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.771150 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-podinfo" pod "c280505e-e32e-4a17-9643-3be7e1250e0d" (UID: "c280505e-e32e-4a17-9643-3be7e1250e0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:35:40.771267 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.771141 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-token" pod "c280505e-e32e-4a17-9643-3be7e1250e0d" (UID: "c280505e-e32e-4a17-9643-3be7e1250e0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:40.870470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.870434 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:40.870470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.870471 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:40.870999 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.870486 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:40.870999 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.870500 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c280505e-e32e-4a17-9643-3be7e1250e0d-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:40.870999 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.870514 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:40.870999 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.870529 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhgmc\" (UniqueName: \"kubernetes.io/projected/c280505e-e32e-4a17-9643-3be7e1250e0d-kube-api-access-jhgmc\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:40.870999 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:40.870543 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c280505e-e32e-4a17-9643-3be7e1250e0d-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:41.497500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:41.497415 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27"] Oct 30 22:35:41.606561 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:41.606523 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df"] Oct 30 22:35:41.687023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:41.686990 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" Oct 30 22:35:41.687023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:41.687016 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5" event={"ID":"c280505e-e32e-4a17-9643-3be7e1250e0d","Type":"ContainerDied","Data":"f9fff5d9bc1e214eb7559e1c7338d80f49aa2aa60a55d81111bbe9ff73477ef9"} Oct 30 22:35:41.687296 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:41.687076 4335 scope.go:122] "RemoveContainer" containerID="427e922cc7671ba9a8af3d009d4cb7fc8bc4c650a187554c0c9258a7373d07ba" Oct 30 22:35:41.908601 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:41.908564 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5"] Oct 30 22:35:41.918022 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:41.917992 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-32dd8d-openshift-default-9f4479c8c-6l4j5"] Oct 30 22:35:43.057635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:43.057574 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c280505e-e32e-4a17-9643-3be7e1250e0d" path="/var/lib/kubelet/pods/c280505e-e32e-4a17-9643-3be7e1250e0d/volumes" Oct 30 22:35:46.498342 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:46.498300 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" podUID="32706948-1fdd-41a7-91a0-4b7b7a1db425" containerName="payload-processing" containerID="cri-o://cd0e42ecf443a2ac8f6f4b3cb5a0da546dd83bf6b7dc34ca3efa377b718c8858" gracePeriod=25 Oct 30 22:35:46.607510 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:46.607452 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" podUID="14fc164d-631a-47ae-8772-162c3688aec0" containerName="payload-pre-processing" containerID="cri-o://95e99977e022d1248f3fac90459e9fdbb622df42701f6ec7f4c69c3b42919803" gracePeriod=25 Oct 30 22:35:46.798362 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:46.798289 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb"] Oct 30 22:35:49.424557 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.424519 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj"] Oct 30 22:35:49.425168 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.425147 4335 cpu_manager.go:414] "containerMap: removing container" podUID="c280505e-e32e-4a17-9643-3be7e1250e0d" containerName="istio-proxy" Oct 30 22:35:49.425275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.425170 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c280505e-e32e-4a17-9643-3be7e1250e0d" containerName="istio-proxy" Oct 30 22:35:49.425275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.425202 4335 cpu_manager.go:414] "containerMap: removing container" podUID="c280505e-e32e-4a17-9643-3be7e1250e0d" containerName="istio-proxy" Oct 30 22:35:49.425275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.425210 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c280505e-e32e-4a17-9643-3be7e1250e0d" containerName="istio-proxy" Oct 30 22:35:49.425275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.425234 4335 cpu_manager.go:414] "containerMap: removing container" podUID="57af8388-c78d-4617-b19a-2adfc7d14e11" containerName="revoke-keys" Oct 30 22:35:49.425275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.425243 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57af8388-c78d-4617-b19a-2adfc7d14e11" containerName="revoke-keys" Oct 30 22:35:49.425550 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.425368 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="c280505e-e32e-4a17-9643-3be7e1250e0d" containerName="istio-proxy" Oct 30 22:35:49.425550 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.425386 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="c280505e-e32e-4a17-9643-3be7e1250e0d" containerName="istio-proxy" Oct 30 22:35:49.425550 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.425397 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="57af8388-c78d-4617-b19a-2adfc7d14e11" containerName="revoke-keys" Oct 30 22:35:49.430401 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.430379 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj" Oct 30 22:35:49.485722 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.485699 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj"] Oct 30 22:35:49.532103 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.532076 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:35:49.532254 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.532215 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:35:49.532327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.532272 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:35:49.540223 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.540201 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:35:49.548639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.548595 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8e93b-8f10-4451-bab6-043704da23ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3367206a-g9pzj\" (UID: \"5fc8e93b-8f10-4451-bab6-043704da23ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj" Oct 30 22:35:49.548734 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.548718 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc8e93b-8f10-4451-bab6-043704da23ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3367206a-g9pzj\" (UID: \"5fc8e93b-8f10-4451-bab6-043704da23ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj" Oct 30 22:35:49.548774 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.548756 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rk4q\" (UniqueName: \"kubernetes.io/projected/5fc8e93b-8f10-4451-bab6-043704da23ef-kube-api-access-6rk4q\") pod \"maas-api-revoke-sub-3367206a-g9pzj\" (UID: \"5fc8e93b-8f10-4451-bab6-043704da23ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj" Oct 30 22:35:49.650211 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.650184 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8e93b-8f10-4451-bab6-043704da23ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3367206a-g9pzj\" (UID: \"5fc8e93b-8f10-4451-bab6-043704da23ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj" Oct 30 22:35:49.650374 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.650251 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc8e93b-8f10-4451-bab6-043704da23ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3367206a-g9pzj\" (UID: \"5fc8e93b-8f10-4451-bab6-043704da23ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj" Oct 30 22:35:49.650374 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.650302 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rk4q\" (UniqueName: \"kubernetes.io/projected/5fc8e93b-8f10-4451-bab6-043704da23ef-kube-api-access-6rk4q\") pod \"maas-api-revoke-sub-3367206a-g9pzj\" (UID: \"5fc8e93b-8f10-4451-bab6-043704da23ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj" Oct 30 22:35:49.651054 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.650963 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8e93b-8f10-4451-bab6-043704da23ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3367206a-g9pzj\" (UID: \"5fc8e93b-8f10-4451-bab6-043704da23ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj" Oct 30 22:35:49.652898 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.652872 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc8e93b-8f10-4451-bab6-043704da23ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3367206a-g9pzj\" (UID: \"5fc8e93b-8f10-4451-bab6-043704da23ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj" Oct 30 22:35:49.711170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.711113 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rk4q\" (UniqueName: \"kubernetes.io/projected/5fc8e93b-8f10-4451-bab6-043704da23ef-kube-api-access-6rk4q\") pod \"maas-api-revoke-sub-3367206a-g9pzj\" (UID: \"5fc8e93b-8f10-4451-bab6-043704da23ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj" Oct 30 22:35:49.743956 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:49.743935 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj" Oct 30 22:35:50.121270 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:50.121249 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj"] Oct 30 22:35:50.123031 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:35:50.123006 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc8e93b_8f10_4451_bab6_043704da23ef.slice/crio-436d26fa67afc700850bfc1d77f6578a20e44b3328b05b700c30941673d5c07b WatchSource:0}: Error finding container 436d26fa67afc700850bfc1d77f6578a20e44b3328b05b700c30941673d5c07b: Status 404 returned error can't find the container with id 436d26fa67afc700850bfc1d77f6578a20e44b3328b05b700c30941673d5c07b Oct 30 22:35:50.679270 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:50.679197 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" containerID="cri-o://653777e7949acc4d0a115d3dcd131cda3c5b0748b43019ea39de612004351bb8" gracePeriod=45 Oct 30 22:35:50.747904 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:50.747876 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj" event={"ID":"5fc8e93b-8f10-4451-bab6-043704da23ef","Type":"ContainerStarted","Data":"7bc42aaad84acb70501b1903ba406ebdcd88ca9022c41f3247a598f11b39cf84"} Oct 30 22:35:50.748058 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:50.747925 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj" event={"ID":"5fc8e93b-8f10-4451-bab6-043704da23ef","Type":"ContainerStarted","Data":"436d26fa67afc700850bfc1d77f6578a20e44b3328b05b700c30941673d5c07b"} Oct 30 22:35:50.866477 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:50.866430 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj" podStartSLOduration=1.866415544 podStartE2EDuration="1.866415544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:50.862887077 +0000 UTC m=+3083.963318810" watchObservedRunningTime="2026-10-30 22:35:50.866415544 +0000 UTC m=+3083.966847273" Oct 30 22:35:50.945556 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:50.945484 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:35:51.063783 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.063753 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-model-cache\") pod \"9975ba00-5118-4e3f-984b-c6621e3c4f77\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " Oct 30 22:35:51.063960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.063810 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-home\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-home\") pod \"9975ba00-5118-4e3f-984b-c6621e3c4f77\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " Oct 30 22:35:51.063960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.063857 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-tmp-dir\") pod \"9975ba00-5118-4e3f-984b-c6621e3c4f77\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " Oct 30 22:35:51.063960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.063946 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9975ba00-5118-4e3f-984b-c6621e3c4f77-tls-certs\" (UniqueName: \"kubernetes.io/secret/9975ba00-5118-4e3f-984b-c6621e3c4f77-tls-certs\") pod \"9975ba00-5118-4e3f-984b-c6621e3c4f77\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " Oct 30 22:35:51.064097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.063978 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9975ba00-5118-4e3f-984b-c6621e3c4f77-kube-api-access-q6gmw\" (UniqueName: \"kubernetes.io/projected/9975ba00-5118-4e3f-984b-c6621e3c4f77-kube-api-access-q6gmw\") pod \"9975ba00-5118-4e3f-984b-c6621e3c4f77\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " Oct 30 22:35:51.064097 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.064024 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-dshm\") pod \"9975ba00-5118-4e3f-984b-c6621e3c4f77\" (UID: \"9975ba00-5118-4e3f-984b-c6621e3c4f77\") " Oct 30 22:35:51.065083 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.065057 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-model-cache" pod "9975ba00-5118-4e3f-984b-c6621e3c4f77" (UID: "9975ba00-5118-4e3f-984b-c6621e3c4f77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:51.065218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.065200 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-home" pod "9975ba00-5118-4e3f-984b-c6621e3c4f77" (UID: "9975ba00-5118-4e3f-984b-c6621e3c4f77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:51.065332 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.065317 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-tmp-dir" pod "9975ba00-5118-4e3f-984b-c6621e3c4f77" (UID: "9975ba00-5118-4e3f-984b-c6621e3c4f77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:51.066777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.066753 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-dshm" pod "9975ba00-5118-4e3f-984b-c6621e3c4f77" (UID: "9975ba00-5118-4e3f-984b-c6621e3c4f77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:51.067320 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.067298 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9975ba00-5118-4e3f-984b-c6621e3c4f77-tls-certs" pod "9975ba00-5118-4e3f-984b-c6621e3c4f77" (UID: "9975ba00-5118-4e3f-984b-c6621e3c4f77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:51.067864 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.067827 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9975ba00-5118-4e3f-984b-c6621e3c4f77-kube-api-access-q6gmw" pod "9975ba00-5118-4e3f-984b-c6621e3c4f77" (UID: "9975ba00-5118-4e3f-984b-c6621e3c4f77"). InnerVolumeSpecName "kube-api-access-q6gmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:51.165727 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.165701 4335 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-model-cache\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:51.165727 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.165731 4335 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-home\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:51.165926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.165742 4335 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-tmp-dir\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:51.165926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.165751 4335 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9975ba00-5118-4e3f-984b-c6621e3c4f77-tls-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:51.165926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.165760 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6gmw\" (UniqueName: \"kubernetes.io/projected/9975ba00-5118-4e3f-984b-c6621e3c4f77-kube-api-access-q6gmw\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:51.165926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.165770 4335 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9975ba00-5118-4e3f-984b-c6621e3c4f77-dshm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:51.756161 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.756132 4335 generic.go:356] "Generic (PLEG): container finished" podID="5fc8e93b-8f10-4451-bab6-043704da23ef" containerID="7bc42aaad84acb70501b1903ba406ebdcd88ca9022c41f3247a598f11b39cf84" exitCode=0 Oct 30 22:35:51.756647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.756224 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj" event={"ID":"5fc8e93b-8f10-4451-bab6-043704da23ef","Type":"ContainerDied","Data":"7bc42aaad84acb70501b1903ba406ebdcd88ca9022c41f3247a598f11b39cf84"} Oct 30 22:35:51.757497 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.757473 4335 generic.go:356] "Generic (PLEG): container finished" podID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerID="653777e7949acc4d0a115d3dcd131cda3c5b0748b43019ea39de612004351bb8" exitCode=0 Oct 30 22:35:51.757643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.757502 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" event={"ID":"9975ba00-5118-4e3f-984b-c6621e3c4f77","Type":"ContainerDied","Data":"653777e7949acc4d0a115d3dcd131cda3c5b0748b43019ea39de612004351bb8"} Oct 30 22:35:51.757643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.757520 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" Oct 30 22:35:51.757643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.757550 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68" event={"ID":"9975ba00-5118-4e3f-984b-c6621e3c4f77","Type":"ContainerDied","Data":"78e3ea3fcd899cabf467f7abee3f1931cf4312c71a3e7a2713ddff03931e6dcc"} Oct 30 22:35:51.757643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.757574 4335 scope.go:122] "RemoveContainer" containerID="653777e7949acc4d0a115d3dcd131cda3c5b0748b43019ea39de612004351bb8" Oct 30 22:35:51.772316 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.772299 4335 scope.go:122] "RemoveContainer" containerID="653777e7949acc4d0a115d3dcd131cda3c5b0748b43019ea39de612004351bb8" Oct 30 22:35:51.772573 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:35:51.772557 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653777e7949acc4d0a115d3dcd131cda3c5b0748b43019ea39de612004351bb8\": container with ID starting with 653777e7949acc4d0a115d3dcd131cda3c5b0748b43019ea39de612004351bb8 not found: ID does not exist" containerID="653777e7949acc4d0a115d3dcd131cda3c5b0748b43019ea39de612004351bb8" Oct 30 22:35:51.772654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.772581 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653777e7949acc4d0a115d3dcd131cda3c5b0748b43019ea39de612004351bb8"} err="failed to get container status \"653777e7949acc4d0a115d3dcd131cda3c5b0748b43019ea39de612004351bb8\": rpc error: code = NotFound desc = could not find container \"653777e7949acc4d0a115d3dcd131cda3c5b0748b43019ea39de612004351bb8\": container with ID starting with 653777e7949acc4d0a115d3dcd131cda3c5b0748b43019ea39de612004351bb8 not found: ID does not exist" Oct 30 22:35:51.799265 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.799238 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb" podUID="833aa422-6f85-413c-ba5e-1befd25b974d" containerName="payload-processing" containerID="cri-o://5a9bbc83abc1f670d5de592cea2db53b4bd1262751da338b64e4d1d3eff68362" gracePeriod=25 Oct 30 22:35:51.985479 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.985448 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68"] Oct 30 22:35:51.990285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:51.990254 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-b9fbe2/premium-w6-b9fbe2-kserve-54dcd98ccd-6jv68"] Oct 30 22:35:52.047597 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.047573 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb" Oct 30 22:35:52.174433 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.174370 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/833aa422-6f85-413c-ba5e-1befd25b974d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833aa422-6f85-413c-ba5e-1befd25b974d-plugins-config-volume\") pod \"833aa422-6f85-413c-ba5e-1befd25b974d\" (UID: \"833aa422-6f85-413c-ba5e-1befd25b974d\") " Oct 30 22:35:52.174593 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.174438 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/833aa422-6f85-413c-ba5e-1befd25b974d-kube-api-access-mq4ct\" (UniqueName: \"kubernetes.io/projected/833aa422-6f85-413c-ba5e-1befd25b974d-kube-api-access-mq4ct\") pod \"833aa422-6f85-413c-ba5e-1befd25b974d\" (UID: \"833aa422-6f85-413c-ba5e-1befd25b974d\") " Oct 30 22:35:52.174895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.174866 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833aa422-6f85-413c-ba5e-1befd25b974d-plugins-config-volume" pod "833aa422-6f85-413c-ba5e-1befd25b974d" (UID: "833aa422-6f85-413c-ba5e-1befd25b974d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:52.177044 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.177016 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833aa422-6f85-413c-ba5e-1befd25b974d-kube-api-access-mq4ct" pod "833aa422-6f85-413c-ba5e-1befd25b974d" (UID: "833aa422-6f85-413c-ba5e-1befd25b974d"). InnerVolumeSpecName "kube-api-access-mq4ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:52.276035 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.276010 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833aa422-6f85-413c-ba5e-1befd25b974d-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:52.276035 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.276034 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq4ct\" (UniqueName: \"kubernetes.io/projected/833aa422-6f85-413c-ba5e-1befd25b974d-kube-api-access-mq4ct\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:52.765171 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.765140 4335 generic.go:356] "Generic (PLEG): container finished" podID="833aa422-6f85-413c-ba5e-1befd25b974d" containerID="5a9bbc83abc1f670d5de592cea2db53b4bd1262751da338b64e4d1d3eff68362" exitCode=0 Oct 30 22:35:52.765642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.765185 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb" Oct 30 22:35:52.765642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.765217 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb" event={"ID":"833aa422-6f85-413c-ba5e-1befd25b974d","Type":"ContainerDied","Data":"5a9bbc83abc1f670d5de592cea2db53b4bd1262751da338b64e4d1d3eff68362"} Oct 30 22:35:52.765642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.765275 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb" event={"ID":"833aa422-6f85-413c-ba5e-1befd25b974d","Type":"ContainerDied","Data":"b3e632ea94fb6f909f71f89b00c396f55c068d06f36d3151c0393e3d70a15da1"} Oct 30 22:35:52.765642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.765298 4335 scope.go:122] "RemoveContainer" containerID="5a9bbc83abc1f670d5de592cea2db53b4bd1262751da338b64e4d1d3eff68362" Oct 30 22:35:52.776295 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.776271 4335 scope.go:122] "RemoveContainer" containerID="5a9bbc83abc1f670d5de592cea2db53b4bd1262751da338b64e4d1d3eff68362" Oct 30 22:35:52.776587 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:35:52.776565 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a9bbc83abc1f670d5de592cea2db53b4bd1262751da338b64e4d1d3eff68362\": container with ID starting with 5a9bbc83abc1f670d5de592cea2db53b4bd1262751da338b64e4d1d3eff68362 not found: ID does not exist" containerID="5a9bbc83abc1f670d5de592cea2db53b4bd1262751da338b64e4d1d3eff68362" Oct 30 22:35:52.776690 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.776596 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9bbc83abc1f670d5de592cea2db53b4bd1262751da338b64e4d1d3eff68362"} err="failed to get container status \"5a9bbc83abc1f670d5de592cea2db53b4bd1262751da338b64e4d1d3eff68362\": rpc error: code = NotFound desc = could not find container \"5a9bbc83abc1f670d5de592cea2db53b4bd1262751da338b64e4d1d3eff68362\": container with ID starting with 5a9bbc83abc1f670d5de592cea2db53b4bd1262751da338b64e4d1d3eff68362 not found: ID does not exist" Oct 30 22:35:52.899218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.899194 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj" Oct 30 22:35:52.930498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.930472 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb"] Oct 30 22:35:52.936039 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.936014 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-55beb8-69986bdd47-xm8fb"] Oct 30 22:35:52.982102 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.982073 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fc8e93b-8f10-4451-bab6-043704da23ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc8e93b-8f10-4451-bab6-043704da23ef-maas-api-xks-ca\") pod \"5fc8e93b-8f10-4451-bab6-043704da23ef\" (UID: \"5fc8e93b-8f10-4451-bab6-043704da23ef\") " Oct 30 22:35:52.982246 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.982160 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fc8e93b-8f10-4451-bab6-043704da23ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8e93b-8f10-4451-bab6-043704da23ef-maas-api-openshift-ca\") pod \"5fc8e93b-8f10-4451-bab6-043704da23ef\" (UID: \"5fc8e93b-8f10-4451-bab6-043704da23ef\") " Oct 30 22:35:52.982246 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.982234 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc8e93b-8f10-4451-bab6-043704da23ef-kube-api-access-6rk4q\" (UniqueName: \"kubernetes.io/projected/5fc8e93b-8f10-4451-bab6-043704da23ef-kube-api-access-6rk4q\") pod \"5fc8e93b-8f10-4451-bab6-043704da23ef\" (UID: \"5fc8e93b-8f10-4451-bab6-043704da23ef\") " Oct 30 22:35:52.982520 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.982495 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc8e93b-8f10-4451-bab6-043704da23ef-maas-api-openshift-ca" pod "5fc8e93b-8f10-4451-bab6-043704da23ef" (UID: "5fc8e93b-8f10-4451-bab6-043704da23ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:52.984392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.984356 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc8e93b-8f10-4451-bab6-043704da23ef-kube-api-access-6rk4q" pod "5fc8e93b-8f10-4451-bab6-043704da23ef" (UID: "5fc8e93b-8f10-4451-bab6-043704da23ef"). InnerVolumeSpecName "kube-api-access-6rk4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:52.984496 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:52.984450 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fc8e93b-8f10-4451-bab6-043704da23ef-maas-api-xks-ca" pod "5fc8e93b-8f10-4451-bab6-043704da23ef" (UID: "5fc8e93b-8f10-4451-bab6-043704da23ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:53.059407 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:53.059379 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="833aa422-6f85-413c-ba5e-1befd25b974d" path="/var/lib/kubelet/pods/833aa422-6f85-413c-ba5e-1befd25b974d/volumes" Oct 30 22:35:53.059994 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:53.059973 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" path="/var/lib/kubelet/pods/9975ba00-5118-4e3f-984b-c6621e3c4f77/volumes" Oct 30 22:35:53.083889 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:53.083867 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rk4q\" (UniqueName: \"kubernetes.io/projected/5fc8e93b-8f10-4451-bab6-043704da23ef-kube-api-access-6rk4q\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:53.083993 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:53.083892 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc8e93b-8f10-4451-bab6-043704da23ef-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:53.083993 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:53.083901 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8e93b-8f10-4451-bab6-043704da23ef-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:53.775023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:53.774990 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj" event={"ID":"5fc8e93b-8f10-4451-bab6-043704da23ef","Type":"ContainerDied","Data":"436d26fa67afc700850bfc1d77f6578a20e44b3328b05b700c30941673d5c07b"} Oct 30 22:35:53.775023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:53.775019 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj" Oct 30 22:35:53.775576 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:53.775037 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="436d26fa67afc700850bfc1d77f6578a20e44b3328b05b700c30941673d5c07b" Oct 30 22:35:55.320928 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:35:55.320892 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:35:55.568791 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:55.568759 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:35:55.997126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:55.997089 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" podUID="32706948-1fdd-41a7-91a0-4b7b7a1db425" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:35:55.997307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:35:55.997127 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" podUID="14fc164d-631a-47ae-8772-162c3688aec0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:36:05.996740 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:05.996701 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" podUID="14fc164d-631a-47ae-8772-162c3688aec0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:36:05.996740 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:05.996716 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" podUID="32706948-1fdd-41a7-91a0-4b7b7a1db425" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:36:06.495145 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.495110 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q"] Oct 30 22:36:06.544271 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.544243 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-55beb8-md94q"] Oct 30 22:36:06.754971 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.754921 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" Oct 30 22:36:06.854462 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.854437 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" Oct 30 22:36:06.856604 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.856579 4335 generic.go:356] "Generic (PLEG): container finished" podID="14fc164d-631a-47ae-8772-162c3688aec0" containerID="95e99977e022d1248f3fac90459e9fdbb622df42701f6ec7f4c69c3b42919803" exitCode=0 Oct 30 22:36:06.856768 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.856632 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" Oct 30 22:36:06.856768 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.856654 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" event={"ID":"14fc164d-631a-47ae-8772-162c3688aec0","Type":"ContainerDied","Data":"95e99977e022d1248f3fac90459e9fdbb622df42701f6ec7f4c69c3b42919803"} Oct 30 22:36:06.856768 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.856719 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df" event={"ID":"14fc164d-631a-47ae-8772-162c3688aec0","Type":"ContainerDied","Data":"8865c28f96cb38dc4f47db3a8425536bdae33aaa104d30687bb4d1aef6f4bf0a"} Oct 30 22:36:06.856768 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.856744 4335 scope.go:122] "RemoveContainer" containerID="95e99977e022d1248f3fac90459e9fdbb622df42701f6ec7f4c69c3b42919803" Oct 30 22:36:06.858003 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.857983 4335 generic.go:356] "Generic (PLEG): container finished" podID="32706948-1fdd-41a7-91a0-4b7b7a1db425" containerID="cd0e42ecf443a2ac8f6f4b3cb5a0da546dd83bf6b7dc34ca3efa377b718c8858" exitCode=0 Oct 30 22:36:06.858131 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.858028 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" Oct 30 22:36:06.858131 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.858109 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" event={"ID":"32706948-1fdd-41a7-91a0-4b7b7a1db425","Type":"ContainerDied","Data":"cd0e42ecf443a2ac8f6f4b3cb5a0da546dd83bf6b7dc34ca3efa377b718c8858"} Oct 30 22:36:06.858267 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.858146 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27" event={"ID":"32706948-1fdd-41a7-91a0-4b7b7a1db425","Type":"ContainerDied","Data":"91e0e6c686575eb529c09fea0d217d264dfae011bc7664f7061d8570ce3f5e67"} Oct 30 22:36:06.866880 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.866862 4335 scope.go:122] "RemoveContainer" containerID="95e99977e022d1248f3fac90459e9fdbb622df42701f6ec7f4c69c3b42919803" Oct 30 22:36:06.867141 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:36:06.867121 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e99977e022d1248f3fac90459e9fdbb622df42701f6ec7f4c69c3b42919803\": container with ID starting with 95e99977e022d1248f3fac90459e9fdbb622df42701f6ec7f4c69c3b42919803 not found: ID does not exist" containerID="95e99977e022d1248f3fac90459e9fdbb622df42701f6ec7f4c69c3b42919803" Oct 30 22:36:06.867196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.867148 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e99977e022d1248f3fac90459e9fdbb622df42701f6ec7f4c69c3b42919803"} err="failed to get container status \"95e99977e022d1248f3fac90459e9fdbb622df42701f6ec7f4c69c3b42919803\": rpc error: code = NotFound desc = could not find container \"95e99977e022d1248f3fac90459e9fdbb622df42701f6ec7f4c69c3b42919803\": container with ID starting with 95e99977e022d1248f3fac90459e9fdbb622df42701f6ec7f4c69c3b42919803 not found: ID does not exist" Oct 30 22:36:06.867196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.867163 4335 scope.go:122] "RemoveContainer" containerID="cd0e42ecf443a2ac8f6f4b3cb5a0da546dd83bf6b7dc34ca3efa377b718c8858" Oct 30 22:36:06.876364 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.876005 4335 scope.go:122] "RemoveContainer" containerID="cd0e42ecf443a2ac8f6f4b3cb5a0da546dd83bf6b7dc34ca3efa377b718c8858" Oct 30 22:36:06.876364 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:36:06.876307 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0e42ecf443a2ac8f6f4b3cb5a0da546dd83bf6b7dc34ca3efa377b718c8858\": container with ID starting with cd0e42ecf443a2ac8f6f4b3cb5a0da546dd83bf6b7dc34ca3efa377b718c8858 not found: ID does not exist" containerID="cd0e42ecf443a2ac8f6f4b3cb5a0da546dd83bf6b7dc34ca3efa377b718c8858" Oct 30 22:36:06.876364 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.876328 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0e42ecf443a2ac8f6f4b3cb5a0da546dd83bf6b7dc34ca3efa377b718c8858"} err="failed to get container status \"cd0e42ecf443a2ac8f6f4b3cb5a0da546dd83bf6b7dc34ca3efa377b718c8858\": rpc error: code = NotFound desc = could not find container \"cd0e42ecf443a2ac8f6f4b3cb5a0da546dd83bf6b7dc34ca3efa377b718c8858\": container with ID starting with cd0e42ecf443a2ac8f6f4b3cb5a0da546dd83bf6b7dc34ca3efa377b718c8858 not found: ID does not exist" Oct 30 22:36:06.902793 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.902773 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32706948-1fdd-41a7-91a0-4b7b7a1db425-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32706948-1fdd-41a7-91a0-4b7b7a1db425-plugins-config-volume\") pod \"32706948-1fdd-41a7-91a0-4b7b7a1db425\" (UID: \"32706948-1fdd-41a7-91a0-4b7b7a1db425\") " Oct 30 22:36:06.902920 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.902833 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32706948-1fdd-41a7-91a0-4b7b7a1db425-kube-api-access-4k2mx\" (UniqueName: \"kubernetes.io/projected/32706948-1fdd-41a7-91a0-4b7b7a1db425-kube-api-access-4k2mx\") pod \"32706948-1fdd-41a7-91a0-4b7b7a1db425\" (UID: \"32706948-1fdd-41a7-91a0-4b7b7a1db425\") " Oct 30 22:36:06.903193 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.903169 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32706948-1fdd-41a7-91a0-4b7b7a1db425-plugins-config-volume" pod "32706948-1fdd-41a7-91a0-4b7b7a1db425" (UID: "32706948-1fdd-41a7-91a0-4b7b7a1db425"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:06.905118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:06.905095 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32706948-1fdd-41a7-91a0-4b7b7a1db425-kube-api-access-4k2mx" pod "32706948-1fdd-41a7-91a0-4b7b7a1db425" (UID: "32706948-1fdd-41a7-91a0-4b7b7a1db425"). InnerVolumeSpecName "kube-api-access-4k2mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:07.003570 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:07.003549 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14fc164d-631a-47ae-8772-162c3688aec0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fc164d-631a-47ae-8772-162c3688aec0-plugins-config-volume\") pod \"14fc164d-631a-47ae-8772-162c3688aec0\" (UID: \"14fc164d-631a-47ae-8772-162c3688aec0\") " Oct 30 22:36:07.003972 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:07.003679 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14fc164d-631a-47ae-8772-162c3688aec0-kube-api-access-ff74v\" (UniqueName: \"kubernetes.io/projected/14fc164d-631a-47ae-8772-162c3688aec0-kube-api-access-ff74v\") pod \"14fc164d-631a-47ae-8772-162c3688aec0\" (UID: \"14fc164d-631a-47ae-8772-162c3688aec0\") " Oct 30 22:36:07.003972 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:07.003878 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32706948-1fdd-41a7-91a0-4b7b7a1db425-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.003972 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:07.003889 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k2mx\" (UniqueName: \"kubernetes.io/projected/32706948-1fdd-41a7-91a0-4b7b7a1db425-kube-api-access-4k2mx\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.003972 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:07.003960 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14fc164d-631a-47ae-8772-162c3688aec0-plugins-config-volume" pod "14fc164d-631a-47ae-8772-162c3688aec0" (UID: "14fc164d-631a-47ae-8772-162c3688aec0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:07.005692 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:07.005652 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14fc164d-631a-47ae-8772-162c3688aec0-kube-api-access-ff74v" pod "14fc164d-631a-47ae-8772-162c3688aec0" (UID: "14fc164d-631a-47ae-8772-162c3688aec0"). InnerVolumeSpecName "kube-api-access-ff74v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:07.058473 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:07.058450 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57af8388-c78d-4617-b19a-2adfc7d14e11" path="/var/lib/kubelet/pods/57af8388-c78d-4617-b19a-2adfc7d14e11/volumes" Oct 30 22:36:07.104867 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:07.104842 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fc164d-631a-47ae-8772-162c3688aec0-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.104867 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:07.104865 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff74v\" (UniqueName: \"kubernetes.io/projected/14fc164d-631a-47ae-8772-162c3688aec0-kube-api-access-ff74v\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.271545 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:07.271484 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df"] Oct 30 22:36:07.279367 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:07.279346 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b9fbe2-6cc7bfdf47-l67df"] Oct 30 22:36:07.388134 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:07.388111 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27"] Oct 30 22:36:07.402526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:07.402503 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b9fbe2-6dfc9f5b6c-xhk27"] Oct 30 22:36:09.056635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:09.056533 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14fc164d-631a-47ae-8772-162c3688aec0" path="/var/lib/kubelet/pods/14fc164d-631a-47ae-8772-162c3688aec0/volumes" Oct 30 22:36:09.057150 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:09.057130 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32706948-1fdd-41a7-91a0-4b7b7a1db425" path="/var/lib/kubelet/pods/32706948-1fdd-41a7-91a0-4b7b7a1db425/volumes" Oct 30 22:36:13.041886 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:13.041847 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7"] Oct 30 22:36:13.089641 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:13.089601 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9fbe2-l92w7"] Oct 30 22:36:15.056423 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:15.056390 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddbae0c7-0910-40f2-bd8c-ef93df5ddee8" path="/var/lib/kubelet/pods/ddbae0c7-0910-40f2-bd8c-ef93df5ddee8/volumes" Oct 30 22:36:15.546139 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:15.546068 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:36:16.324406 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.324377 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9"] Oct 30 22:36:16.324872 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.324855 4335 cpu_manager.go:414] "containerMap: removing container" podUID="14fc164d-631a-47ae-8772-162c3688aec0" containerName="payload-pre-processing" Oct 30 22:36:16.324918 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.324873 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14fc164d-631a-47ae-8772-162c3688aec0" containerName="payload-pre-processing" Oct 30 22:36:16.324918 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.324888 4335 cpu_manager.go:414] "containerMap: removing container" podUID="32706948-1fdd-41a7-91a0-4b7b7a1db425" containerName="payload-processing" Oct 30 22:36:16.324918 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.324893 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32706948-1fdd-41a7-91a0-4b7b7a1db425" containerName="payload-processing" Oct 30 22:36:16.324918 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.324905 4335 cpu_manager.go:414] "containerMap: removing container" podUID="833aa422-6f85-413c-ba5e-1befd25b974d" containerName="payload-processing" Oct 30 22:36:16.324918 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.324910 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833aa422-6f85-413c-ba5e-1befd25b974d" containerName="payload-processing" Oct 30 22:36:16.325089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.324933 4335 cpu_manager.go:414] "containerMap: removing container" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" Oct 30 22:36:16.325089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.324938 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" Oct 30 22:36:16.325089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.324949 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5fc8e93b-8f10-4451-bab6-043704da23ef" containerName="revoke-keys" Oct 30 22:36:16.325089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.324954 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc8e93b-8f10-4451-bab6-043704da23ef" containerName="revoke-keys" Oct 30 22:36:16.325089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.325010 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="32706948-1fdd-41a7-91a0-4b7b7a1db425" containerName="payload-processing" Oct 30 22:36:16.325089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.325020 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="833aa422-6f85-413c-ba5e-1befd25b974d" containerName="payload-processing" Oct 30 22:36:16.325089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.325029 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="9975ba00-5118-4e3f-984b-c6621e3c4f77" containerName="main" Oct 30 22:36:16.325089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.325037 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc8e93b-8f10-4451-bab6-043704da23ef" containerName="revoke-keys" Oct 30 22:36:16.325089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.325042 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="14fc164d-631a-47ae-8772-162c3688aec0" containerName="payload-pre-processing" Oct 30 22:36:16.329658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.329608 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9" Oct 30 22:36:16.381982 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.381958 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg5gn\" (UniqueName: \"kubernetes.io/projected/7091da33-9120-43ab-b4ab-26d3636048a9-kube-api-access-tg5gn\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9\" (UID: \"7091da33-9120-43ab-b4ab-26d3636048a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9" Oct 30 22:36:16.382127 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.381997 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7091da33-9120-43ab-b4ab-26d3636048a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9\" (UID: \"7091da33-9120-43ab-b4ab-26d3636048a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9" Oct 30 22:36:16.382127 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.382082 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7091da33-9120-43ab-b4ab-26d3636048a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9\" (UID: \"7091da33-9120-43ab-b4ab-26d3636048a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9" Oct 30 22:36:16.386341 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.386321 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9"] Oct 30 22:36:16.431669 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.431645 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:36:16.431774 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.431678 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:36:16.447361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.447340 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:36:16.447542 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.447530 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:36:16.482568 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.482543 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg5gn\" (UniqueName: \"kubernetes.io/projected/7091da33-9120-43ab-b4ab-26d3636048a9-kube-api-access-tg5gn\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9\" (UID: \"7091da33-9120-43ab-b4ab-26d3636048a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9" Oct 30 22:36:16.482693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.482585 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7091da33-9120-43ab-b4ab-26d3636048a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9\" (UID: \"7091da33-9120-43ab-b4ab-26d3636048a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9" Oct 30 22:36:16.482693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.482635 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7091da33-9120-43ab-b4ab-26d3636048a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9\" (UID: \"7091da33-9120-43ab-b4ab-26d3636048a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9" Oct 30 22:36:16.483189 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.483166 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7091da33-9120-43ab-b4ab-26d3636048a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9\" (UID: \"7091da33-9120-43ab-b4ab-26d3636048a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9" Oct 30 22:36:16.485246 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.485221 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7091da33-9120-43ab-b4ab-26d3636048a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9\" (UID: \"7091da33-9120-43ab-b4ab-26d3636048a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9" Oct 30 22:36:16.542935 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.542914 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg5gn\" (UniqueName: \"kubernetes.io/projected/7091da33-9120-43ab-b4ab-26d3636048a9-kube-api-access-tg5gn\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9\" (UID: \"7091da33-9120-43ab-b4ab-26d3636048a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9" Oct 30 22:36:16.642272 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:16.642222 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9" Oct 30 22:36:17.024184 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:17.024155 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9"] Oct 30 22:36:17.025193 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:36:17.025158 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7091da33_9120_43ab_b4ab_26d3636048a9.slice/crio-c8a74907337ef7fb3c1e97965894a3cb4113c8d503a78919edf2c1f0a6128780 WatchSource:0}: Error finding container c8a74907337ef7fb3c1e97965894a3cb4113c8d503a78919edf2c1f0a6128780: Status 404 returned error can't find the container with id c8a74907337ef7fb3c1e97965894a3cb4113c8d503a78919edf2c1f0a6128780 Oct 30 22:36:17.947425 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:17.947391 4335 generic.go:356] "Generic (PLEG): container finished" podID="7091da33-9120-43ab-b4ab-26d3636048a9" containerID="26dc134a9e4048ba17efddf5ec5b4293934cd5a514af22c030a69907c44f3da3" exitCode=0 Oct 30 22:36:17.947789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:17.947463 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9" event={"ID":"7091da33-9120-43ab-b4ab-26d3636048a9","Type":"ContainerDied","Data":"26dc134a9e4048ba17efddf5ec5b4293934cd5a514af22c030a69907c44f3da3"} Oct 30 22:36:17.947789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:17.947498 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9" event={"ID":"7091da33-9120-43ab-b4ab-26d3636048a9","Type":"ContainerStarted","Data":"c8a74907337ef7fb3c1e97965894a3cb4113c8d503a78919edf2c1f0a6128780"} Oct 30 22:36:19.101856 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:19.101831 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9" Oct 30 22:36:19.207408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:19.207353 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7091da33-9120-43ab-b4ab-26d3636048a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7091da33-9120-43ab-b4ab-26d3636048a9-maas-api-openshift-ca\") pod \"7091da33-9120-43ab-b4ab-26d3636048a9\" (UID: \"7091da33-9120-43ab-b4ab-26d3636048a9\") " Oct 30 22:36:19.207408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:19.207390 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7091da33-9120-43ab-b4ab-26d3636048a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7091da33-9120-43ab-b4ab-26d3636048a9-maas-api-xks-ca\") pod \"7091da33-9120-43ab-b4ab-26d3636048a9\" (UID: \"7091da33-9120-43ab-b4ab-26d3636048a9\") " Oct 30 22:36:19.207558 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:19.207448 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7091da33-9120-43ab-b4ab-26d3636048a9-kube-api-access-tg5gn\" (UniqueName: \"kubernetes.io/projected/7091da33-9120-43ab-b4ab-26d3636048a9-kube-api-access-tg5gn\") pod \"7091da33-9120-43ab-b4ab-26d3636048a9\" (UID: \"7091da33-9120-43ab-b4ab-26d3636048a9\") " Oct 30 22:36:19.207722 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:19.207700 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7091da33-9120-43ab-b4ab-26d3636048a9-maas-api-openshift-ca" pod "7091da33-9120-43ab-b4ab-26d3636048a9" (UID: "7091da33-9120-43ab-b4ab-26d3636048a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:19.207818 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:19.207800 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7091da33-9120-43ab-b4ab-26d3636048a9-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:19.209665 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:19.209646 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7091da33-9120-43ab-b4ab-26d3636048a9-kube-api-access-tg5gn" pod "7091da33-9120-43ab-b4ab-26d3636048a9" (UID: "7091da33-9120-43ab-b4ab-26d3636048a9"). InnerVolumeSpecName "kube-api-access-tg5gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:19.209753 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:19.209737 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7091da33-9120-43ab-b4ab-26d3636048a9-maas-api-xks-ca" pod "7091da33-9120-43ab-b4ab-26d3636048a9" (UID: "7091da33-9120-43ab-b4ab-26d3636048a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:19.309088 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:19.309058 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7091da33-9120-43ab-b4ab-26d3636048a9-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:19.309088 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:19.309090 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg5gn\" (UniqueName: \"kubernetes.io/projected/7091da33-9120-43ab-b4ab-26d3636048a9-kube-api-access-tg5gn\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:19.961222 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:19.961184 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9" event={"ID":"7091da33-9120-43ab-b4ab-26d3636048a9","Type":"ContainerDied","Data":"c8a74907337ef7fb3c1e97965894a3cb4113c8d503a78919edf2c1f0a6128780"} Oct 30 22:36:19.961383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:19.961236 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9" Oct 30 22:36:19.961383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:19.961264 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8a74907337ef7fb3c1e97965894a3cb4113c8d503a78919edf2c1f0a6128780" Oct 30 22:36:25.320742 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:36:25.320697 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:36:39.387180 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:39.387143 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9"] Oct 30 22:36:39.436353 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:39.436322 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d1ca4b-g5qd9"] Oct 30 22:36:41.056695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:41.056664 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7091da33-9120-43ab-b4ab-26d3636048a9" path="/var/lib/kubelet/pods/7091da33-9120-43ab-b4ab-26d3636048a9/volumes" Oct 30 22:36:42.816222 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:42.816182 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:36:55.320352 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:36:55.320315 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:36:57.585564 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.585529 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk"] Oct 30 22:36:57.586040 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.585956 4335 cpu_manager.go:414] "containerMap: removing container" podUID="7091da33-9120-43ab-b4ab-26d3636048a9" containerName="revoke-keys" Oct 30 22:36:57.586040 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.585967 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7091da33-9120-43ab-b4ab-26d3636048a9" containerName="revoke-keys" Oct 30 22:36:57.586040 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.586035 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="7091da33-9120-43ab-b4ab-26d3636048a9" containerName="revoke-keys" Oct 30 22:36:57.590867 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.590849 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk" Oct 30 22:36:57.644058 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.644017 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:36:57.644058 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.644042 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:36:57.693221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.693198 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:36:57.693560 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.693543 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:36:57.697009 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.696985 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk"] Oct 30 22:36:57.756264 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.756222 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scdp2\" (UniqueName: \"kubernetes.io/projected/71ada944-71f3-45f1-8311-70137f5a862b-kube-api-access-scdp2\") pod \"maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk\" (UID: \"71ada944-71f3-45f1-8311-70137f5a862b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk" Oct 30 22:36:57.756426 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.756298 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ada944-71f3-45f1-8311-70137f5a862b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk\" (UID: \"71ada944-71f3-45f1-8311-70137f5a862b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk" Oct 30 22:36:57.756426 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.756402 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ada944-71f3-45f1-8311-70137f5a862b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk\" (UID: \"71ada944-71f3-45f1-8311-70137f5a862b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk" Oct 30 22:36:57.857743 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.857674 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ada944-71f3-45f1-8311-70137f5a862b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk\" (UID: \"71ada944-71f3-45f1-8311-70137f5a862b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk" Oct 30 22:36:57.857743 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.857733 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ada944-71f3-45f1-8311-70137f5a862b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk\" (UID: \"71ada944-71f3-45f1-8311-70137f5a862b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk" Oct 30 22:36:57.857949 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.857827 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scdp2\" (UniqueName: \"kubernetes.io/projected/71ada944-71f3-45f1-8311-70137f5a862b-kube-api-access-scdp2\") pod \"maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk\" (UID: \"71ada944-71f3-45f1-8311-70137f5a862b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk" Oct 30 22:36:57.858391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.858371 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ada944-71f3-45f1-8311-70137f5a862b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk\" (UID: \"71ada944-71f3-45f1-8311-70137f5a862b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk" Oct 30 22:36:57.860104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.860083 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ada944-71f3-45f1-8311-70137f5a862b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk\" (UID: \"71ada944-71f3-45f1-8311-70137f5a862b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk" Oct 30 22:36:57.919047 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:57.919017 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scdp2\" (UniqueName: \"kubernetes.io/projected/71ada944-71f3-45f1-8311-70137f5a862b-kube-api-access-scdp2\") pod \"maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk\" (UID: \"71ada944-71f3-45f1-8311-70137f5a862b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk" Oct 30 22:36:58.203384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:58.203313 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk" Oct 30 22:36:58.581863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:58.581837 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk"] Oct 30 22:36:58.583493 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:36:58.583460 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ada944_71f3_45f1_8311_70137f5a862b.slice/crio-deff0e8e9604204f76ea91bee3a2230cecfe11e005436b3beb1f35fc74665a05 WatchSource:0}: Error finding container deff0e8e9604204f76ea91bee3a2230cecfe11e005436b3beb1f35fc74665a05: Status 404 returned error can't find the container with id deff0e8e9604204f76ea91bee3a2230cecfe11e005436b3beb1f35fc74665a05 Oct 30 22:36:59.208441 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:59.208355 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk" event={"ID":"71ada944-71f3-45f1-8311-70137f5a862b","Type":"ContainerStarted","Data":"bb599e3e81daf13e736a8cf9f367fee351e92013a019b7d4de1c38c7db013a2a"} Oct 30 22:36:59.208441 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:59.208419 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk" event={"ID":"71ada944-71f3-45f1-8311-70137f5a862b","Type":"ContainerStarted","Data":"deff0e8e9604204f76ea91bee3a2230cecfe11e005436b3beb1f35fc74665a05"} Oct 30 22:36:59.327010 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:36:59.326952 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk" podStartSLOduration=2.3269392939999998 podStartE2EDuration="2.326939294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:59.324415556 +0000 UTC m=+3152.424847291" watchObservedRunningTime="2026-10-30 22:36:59.326939294 +0000 UTC m=+3152.427371051" Oct 30 22:37:00.216296 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:00.216262 4335 generic.go:356] "Generic (PLEG): container finished" podID="71ada944-71f3-45f1-8311-70137f5a862b" containerID="bb599e3e81daf13e736a8cf9f367fee351e92013a019b7d4de1c38c7db013a2a" exitCode=0 Oct 30 22:37:00.216744 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:00.216325 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk" event={"ID":"71ada944-71f3-45f1-8311-70137f5a862b","Type":"ContainerDied","Data":"bb599e3e81daf13e736a8cf9f367fee351e92013a019b7d4de1c38c7db013a2a"} Oct 30 22:37:01.363946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:01.363924 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk" Oct 30 22:37:01.488693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:01.488588 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ada944-71f3-45f1-8311-70137f5a862b-kube-api-access-scdp2\" (UniqueName: \"kubernetes.io/projected/71ada944-71f3-45f1-8311-70137f5a862b-kube-api-access-scdp2\") pod \"71ada944-71f3-45f1-8311-70137f5a862b\" (UID: \"71ada944-71f3-45f1-8311-70137f5a862b\") " Oct 30 22:37:01.488693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:01.488675 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ada944-71f3-45f1-8311-70137f5a862b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ada944-71f3-45f1-8311-70137f5a862b-maas-api-openshift-ca\") pod \"71ada944-71f3-45f1-8311-70137f5a862b\" (UID: \"71ada944-71f3-45f1-8311-70137f5a862b\") " Oct 30 22:37:01.488898 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:01.488792 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71ada944-71f3-45f1-8311-70137f5a862b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ada944-71f3-45f1-8311-70137f5a862b-maas-api-xks-ca\") pod \"71ada944-71f3-45f1-8311-70137f5a862b\" (UID: \"71ada944-71f3-45f1-8311-70137f5a862b\") " Oct 30 22:37:01.489103 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:01.489072 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ada944-71f3-45f1-8311-70137f5a862b-maas-api-openshift-ca" pod "71ada944-71f3-45f1-8311-70137f5a862b" (UID: "71ada944-71f3-45f1-8311-70137f5a862b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:01.490800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:01.490777 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ada944-71f3-45f1-8311-70137f5a862b-kube-api-access-scdp2" pod "71ada944-71f3-45f1-8311-70137f5a862b" (UID: "71ada944-71f3-45f1-8311-70137f5a862b"). InnerVolumeSpecName "kube-api-access-scdp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:01.490949 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:01.490926 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71ada944-71f3-45f1-8311-70137f5a862b-maas-api-xks-ca" pod "71ada944-71f3-45f1-8311-70137f5a862b" (UID: "71ada944-71f3-45f1-8311-70137f5a862b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:01.589957 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:01.589932 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ada944-71f3-45f1-8311-70137f5a862b-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:01.589957 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:01.589956 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scdp2\" (UniqueName: \"kubernetes.io/projected/71ada944-71f3-45f1-8311-70137f5a862b-kube-api-access-scdp2\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:01.590094 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:01.589966 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ada944-71f3-45f1-8311-70137f5a862b-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:02.048244 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:02.048212 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8"] Oct 30 22:37:02.069045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:02.069020 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8"] Oct 30 22:37:02.098749 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:02.098199 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7"] Oct 30 22:37:02.229846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:02.229816 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk" Oct 30 22:37:02.230005 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:02.229849 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk" event={"ID":"71ada944-71f3-45f1-8311-70137f5a862b","Type":"ContainerDied","Data":"deff0e8e9604204f76ea91bee3a2230cecfe11e005436b3beb1f35fc74665a05"} Oct 30 22:37:02.230005 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:02.229909 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deff0e8e9604204f76ea91bee3a2230cecfe11e005436b3beb1f35fc74665a05" Oct 30 22:37:07.998311 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:07.998283 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j"] Oct 30 22:37:08.751246 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:08.751209 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj"] Oct 30 22:37:08.751723 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:08.751705 4335 cpu_manager.go:414] "containerMap: removing container" podUID="71ada944-71f3-45f1-8311-70137f5a862b" containerName="revoke-keys" Oct 30 22:37:08.751831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:08.751725 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ada944-71f3-45f1-8311-70137f5a862b" containerName="revoke-keys" Oct 30 22:37:08.751902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:08.751836 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ada944-71f3-45f1-8311-70137f5a862b" containerName="revoke-keys" Oct 30 22:37:08.755045 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:08.755024 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj" Oct 30 22:37:08.807300 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:08.807279 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:37:08.809974 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:08.809956 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:37:08.810278 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:08.810263 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:37:08.810390 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:08.810377 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:37:08.813562 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:08.813543 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj"] Oct 30 22:37:08.848844 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:08.848822 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dd4857-0bb8-4daf-b312-2202c1b91795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b97e616a-hrnwj\" (UID: \"65dd4857-0bb8-4daf-b312-2202c1b91795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj" Oct 30 22:37:08.848942 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:08.848857 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dd4857-0bb8-4daf-b312-2202c1b91795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b97e616a-hrnwj\" (UID: \"65dd4857-0bb8-4daf-b312-2202c1b91795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj" Oct 30 22:37:08.849028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:08.849006 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phsm2\" (UniqueName: \"kubernetes.io/projected/65dd4857-0bb8-4daf-b312-2202c1b91795-kube-api-access-phsm2\") pod \"maas-api-revoke-sub-b97e616a-hrnwj\" (UID: \"65dd4857-0bb8-4daf-b312-2202c1b91795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj" Oct 30 22:37:08.949997 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:08.949967 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phsm2\" (UniqueName: \"kubernetes.io/projected/65dd4857-0bb8-4daf-b312-2202c1b91795-kube-api-access-phsm2\") pod \"maas-api-revoke-sub-b97e616a-hrnwj\" (UID: \"65dd4857-0bb8-4daf-b312-2202c1b91795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj" Oct 30 22:37:08.950168 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:08.950036 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dd4857-0bb8-4daf-b312-2202c1b91795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b97e616a-hrnwj\" (UID: \"65dd4857-0bb8-4daf-b312-2202c1b91795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj" Oct 30 22:37:08.950168 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:08.950069 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dd4857-0bb8-4daf-b312-2202c1b91795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b97e616a-hrnwj\" (UID: \"65dd4857-0bb8-4daf-b312-2202c1b91795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj" Oct 30 22:37:08.950694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:08.950674 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dd4857-0bb8-4daf-b312-2202c1b91795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b97e616a-hrnwj\" (UID: \"65dd4857-0bb8-4daf-b312-2202c1b91795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj" Oct 30 22:37:08.952563 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:08.952545 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dd4857-0bb8-4daf-b312-2202c1b91795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b97e616a-hrnwj\" (UID: \"65dd4857-0bb8-4daf-b312-2202c1b91795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj" Oct 30 22:37:09.010833 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:09.010777 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phsm2\" (UniqueName: \"kubernetes.io/projected/65dd4857-0bb8-4daf-b312-2202c1b91795-kube-api-access-phsm2\") pod \"maas-api-revoke-sub-b97e616a-hrnwj\" (UID: \"65dd4857-0bb8-4daf-b312-2202c1b91795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj" Oct 30 22:37:09.068107 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:09.068085 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj" Oct 30 22:37:09.454185 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:09.454150 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj"] Oct 30 22:37:10.281137 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:10.281099 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj" event={"ID":"65dd4857-0bb8-4daf-b312-2202c1b91795","Type":"ContainerStarted","Data":"c36b2ad5e446ad4746f72cd8f26ac97205e33ade178001d1d1359213a814126d"} Oct 30 22:37:10.281588 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:10.281154 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj" event={"ID":"65dd4857-0bb8-4daf-b312-2202c1b91795","Type":"ContainerStarted","Data":"d796a03f825044d38c9877093eac7c03534a3652a2233bca397e74b66cc5c574"} Oct 30 22:37:10.397243 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:10.397200 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj" podStartSLOduration=2.397186592 podStartE2EDuration="2.397186592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:10.396156362 +0000 UTC m=+3163.496588095" watchObservedRunningTime="2026-10-30 22:37:10.397186592 +0000 UTC m=+3163.497618323" Oct 30 22:37:11.288963 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:11.288928 4335 generic.go:356] "Generic (PLEG): container finished" podID="65dd4857-0bb8-4daf-b312-2202c1b91795" containerID="c36b2ad5e446ad4746f72cd8f26ac97205e33ade178001d1d1359213a814126d" exitCode=0 Oct 30 22:37:11.289420 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:11.288993 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj" event={"ID":"65dd4857-0bb8-4daf-b312-2202c1b91795","Type":"ContainerDied","Data":"c36b2ad5e446ad4746f72cd8f26ac97205e33ade178001d1d1359213a814126d"} Oct 30 22:37:12.452056 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:12.452033 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj" Oct 30 22:37:12.483102 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:12.483075 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65dd4857-0bb8-4daf-b312-2202c1b91795-kube-api-access-phsm2\" (UniqueName: \"kubernetes.io/projected/65dd4857-0bb8-4daf-b312-2202c1b91795-kube-api-access-phsm2\") pod \"65dd4857-0bb8-4daf-b312-2202c1b91795\" (UID: \"65dd4857-0bb8-4daf-b312-2202c1b91795\") " Oct 30 22:37:12.483230 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:12.483138 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65dd4857-0bb8-4daf-b312-2202c1b91795-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dd4857-0bb8-4daf-b312-2202c1b91795-maas-api-xks-ca\") pod \"65dd4857-0bb8-4daf-b312-2202c1b91795\" (UID: \"65dd4857-0bb8-4daf-b312-2202c1b91795\") " Oct 30 22:37:12.483365 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:12.483345 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65dd4857-0bb8-4daf-b312-2202c1b91795-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dd4857-0bb8-4daf-b312-2202c1b91795-maas-api-openshift-ca\") pod \"65dd4857-0bb8-4daf-b312-2202c1b91795\" (UID: \"65dd4857-0bb8-4daf-b312-2202c1b91795\") " Oct 30 22:37:12.483706 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:12.483681 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65dd4857-0bb8-4daf-b312-2202c1b91795-maas-api-openshift-ca" pod "65dd4857-0bb8-4daf-b312-2202c1b91795" (UID: "65dd4857-0bb8-4daf-b312-2202c1b91795"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:12.483863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:12.483846 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dd4857-0bb8-4daf-b312-2202c1b91795-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:12.485189 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:12.485170 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65dd4857-0bb8-4daf-b312-2202c1b91795-maas-api-xks-ca" pod "65dd4857-0bb8-4daf-b312-2202c1b91795" (UID: "65dd4857-0bb8-4daf-b312-2202c1b91795"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:12.485429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:12.485405 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65dd4857-0bb8-4daf-b312-2202c1b91795-kube-api-access-phsm2" pod "65dd4857-0bb8-4daf-b312-2202c1b91795" (UID: "65dd4857-0bb8-4daf-b312-2202c1b91795"). InnerVolumeSpecName "kube-api-access-phsm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:12.585232 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:12.585169 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phsm2\" (UniqueName: \"kubernetes.io/projected/65dd4857-0bb8-4daf-b312-2202c1b91795-kube-api-access-phsm2\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:12.585232 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:12.585196 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dd4857-0bb8-4daf-b312-2202c1b91795-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:12.999513 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:12.999447 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" podUID="f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31" containerName="payload-processing" containerID="cri-o://cb07f312897257e9ba1b6d348af6b0c97bf28ba42d2441ae8711e0f747173e3e" gracePeriod=25 Oct 30 22:37:13.305055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:13.305028 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj" Oct 30 22:37:13.305202 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:13.305059 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj" event={"ID":"65dd4857-0bb8-4daf-b312-2202c1b91795","Type":"ContainerDied","Data":"d796a03f825044d38c9877093eac7c03534a3652a2233bca397e74b66cc5c574"} Oct 30 22:37:13.305202 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:13.305107 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d796a03f825044d38c9877093eac7c03534a3652a2233bca397e74b66cc5c574" Oct 30 22:37:17.068547 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.068512 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" containerID="cri-o://5df9f26475e5c5bc2e65c78008d1bdccfa3218ae11c8071770b1cb0b522ef702" gracePeriod=45 Oct 30 22:37:17.091040 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.091005 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" containerID="cri-o://2b2bee240b587aa4b71594af8427567f2e16e17d6315d1425b01bfa35d54f531" gracePeriod=45 Oct 30 22:37:17.123662 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.123606 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" containerID="cri-o://1ed59dcbc635e21756cadafbf5a84a3796ac53e87950eb92b137b3f7cda3682f" gracePeriod=45 Oct 30 22:37:17.330964 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.330896 4335 generic.go:356] "Generic (PLEG): container finished" podID="43393e2f-feb9-4698-8f80-5291391625b3" containerID="2b2bee240b587aa4b71594af8427567f2e16e17d6315d1425b01bfa35d54f531" exitCode=0 Oct 30 22:37:17.331107 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.330972 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" event={"ID":"43393e2f-feb9-4698-8f80-5291391625b3","Type":"ContainerDied","Data":"2b2bee240b587aa4b71594af8427567f2e16e17d6315d1425b01bfa35d54f531"} Oct 30 22:37:17.333008 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.332983 4335 generic.go:356] "Generic (PLEG): container finished" podID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerID="5df9f26475e5c5bc2e65c78008d1bdccfa3218ae11c8071770b1cb0b522ef702" exitCode=0 Oct 30 22:37:17.333149 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.333052 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" event={"ID":"7c4a9040-3e1b-4c8c-b532-36140681da06","Type":"ContainerDied","Data":"5df9f26475e5c5bc2e65c78008d1bdccfa3218ae11c8071770b1cb0b522ef702"} Oct 30 22:37:17.334899 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.334879 4335 generic.go:356] "Generic (PLEG): container finished" podID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerID="1ed59dcbc635e21756cadafbf5a84a3796ac53e87950eb92b137b3f7cda3682f" exitCode=0 Oct 30 22:37:17.335019 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.334932 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" event={"ID":"9a0129d4-b54f-4081-9f28-3bc5c61dbf17","Type":"ContainerDied","Data":"1ed59dcbc635e21756cadafbf5a84a3796ac53e87950eb92b137b3f7cda3682f"} Oct 30 22:37:17.387077 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.387048 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:37:17.405349 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.405327 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:37:17.408255 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.408235 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:37:17.424494 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.424467 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-model-cache\") pod \"7c4a9040-3e1b-4c8c-b532-36140681da06\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " Oct 30 22:37:17.424604 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.424511 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43393e2f-feb9-4698-8f80-5291391625b3-kube-api-access-l25mc\" (UniqueName: \"kubernetes.io/projected/43393e2f-feb9-4698-8f80-5291391625b3-kube-api-access-l25mc\") pod \"43393e2f-feb9-4698-8f80-5291391625b3\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " Oct 30 22:37:17.424604 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.424567 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-tmp-dir\") pod \"43393e2f-feb9-4698-8f80-5291391625b3\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " Oct 30 22:37:17.424769 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.424683 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-model-cache\") pod \"43393e2f-feb9-4698-8f80-5291391625b3\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " Oct 30 22:37:17.424769 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.424727 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-home\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-home\") pod \"7c4a9040-3e1b-4c8c-b532-36140681da06\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " Oct 30 22:37:17.424769 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.424737 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-model-cache" pod "7c4a9040-3e1b-4c8c-b532-36140681da06" (UID: "7c4a9040-3e1b-4c8c-b532-36140681da06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:17.424769 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.424763 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c4a9040-3e1b-4c8c-b532-36140681da06-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4a9040-3e1b-4c8c-b532-36140681da06-tls-certs\") pod \"7c4a9040-3e1b-4c8c-b532-36140681da06\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " Oct 30 22:37:17.424992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.424792 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-dshm\") pod \"7c4a9040-3e1b-4c8c-b532-36140681da06\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " Oct 30 22:37:17.424992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.424825 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43393e2f-feb9-4698-8f80-5291391625b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/43393e2f-feb9-4698-8f80-5291391625b3-tls-certs\") pod \"43393e2f-feb9-4698-8f80-5291391625b3\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " Oct 30 22:37:17.424992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.424856 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-dshm\") pod \"43393e2f-feb9-4698-8f80-5291391625b3\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " Oct 30 22:37:17.424992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.424877 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-tmp-dir" pod "43393e2f-feb9-4698-8f80-5291391625b3" (UID: "43393e2f-feb9-4698-8f80-5291391625b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:17.424992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.424884 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-home\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-home\") pod \"43393e2f-feb9-4698-8f80-5291391625b3\" (UID: \"43393e2f-feb9-4698-8f80-5291391625b3\") " Oct 30 22:37:17.424992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.424912 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-tmp-dir\") pod \"7c4a9040-3e1b-4c8c-b532-36140681da06\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " Oct 30 22:37:17.424992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.424958 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c4a9040-3e1b-4c8c-b532-36140681da06-kube-api-access-wwfwb\" (UniqueName: \"kubernetes.io/projected/7c4a9040-3e1b-4c8c-b532-36140681da06-kube-api-access-wwfwb\") pod \"7c4a9040-3e1b-4c8c-b532-36140681da06\" (UID: \"7c4a9040-3e1b-4c8c-b532-36140681da06\") " Oct 30 22:37:17.424992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.424969 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-home" pod "7c4a9040-3e1b-4c8c-b532-36140681da06" (UID: "7c4a9040-3e1b-4c8c-b532-36140681da06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:17.424992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.424979 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-model-cache" pod "43393e2f-feb9-4698-8f80-5291391625b3" (UID: "43393e2f-feb9-4698-8f80-5291391625b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:17.425456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.425172 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-home" pod "43393e2f-feb9-4698-8f80-5291391625b3" (UID: "43393e2f-feb9-4698-8f80-5291391625b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:17.425456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.425317 4335 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-home\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:17.425456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.425336 4335 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-model-cache\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:17.425456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.425350 4335 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-tmp-dir\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:17.425456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.425364 4335 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-model-cache\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:17.425456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.425379 4335 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-home\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:17.425825 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.425463 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-tmp-dir" pod "7c4a9040-3e1b-4c8c-b532-36140681da06" (UID: "7c4a9040-3e1b-4c8c-b532-36140681da06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:17.427274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.427238 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43393e2f-feb9-4698-8f80-5291391625b3-tls-certs" pod "43393e2f-feb9-4698-8f80-5291391625b3" (UID: "43393e2f-feb9-4698-8f80-5291391625b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:17.427395 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.427372 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43393e2f-feb9-4698-8f80-5291391625b3-kube-api-access-l25mc" pod "43393e2f-feb9-4698-8f80-5291391625b3" (UID: "43393e2f-feb9-4698-8f80-5291391625b3"). InnerVolumeSpecName "kube-api-access-l25mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:17.427466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.427432 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4a9040-3e1b-4c8c-b532-36140681da06-kube-api-access-wwfwb" pod "7c4a9040-3e1b-4c8c-b532-36140681da06" (UID: "7c4a9040-3e1b-4c8c-b532-36140681da06"). InnerVolumeSpecName "kube-api-access-wwfwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:17.427523 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.427501 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c4a9040-3e1b-4c8c-b532-36140681da06-tls-certs" pod "7c4a9040-3e1b-4c8c-b532-36140681da06" (UID: "7c4a9040-3e1b-4c8c-b532-36140681da06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:17.427523 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.427507 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-dshm" pod "43393e2f-feb9-4698-8f80-5291391625b3" (UID: "43393e2f-feb9-4698-8f80-5291391625b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:17.427735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.427719 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-dshm" pod "7c4a9040-3e1b-4c8c-b532-36140681da06" (UID: "7c4a9040-3e1b-4c8c-b532-36140681da06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:17.525759 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.525727 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-model-cache\") pod \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " Oct 30 22:37:17.525977 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.525832 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-kube-api-access-fgrtd\" (UniqueName: \"kubernetes.io/projected/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-kube-api-access-fgrtd\") pod \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " Oct 30 22:37:17.525977 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.525880 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-tmp-dir\") pod \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " Oct 30 22:37:17.526115 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.526026 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-tls-certs\") pod \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " Oct 30 22:37:17.526115 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.526050 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-model-cache" pod "9a0129d4-b54f-4081-9f28-3bc5c61dbf17" (UID: "9a0129d4-b54f-4081-9f28-3bc5c61dbf17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:17.526115 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.526085 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-home\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-home\") pod \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " Oct 30 22:37:17.526267 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.526128 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-dshm\") pod \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\" (UID: \"9a0129d4-b54f-4081-9f28-3bc5c61dbf17\") " Oct 30 22:37:17.526531 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.526508 4335 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-tmp-dir\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:17.526766 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.526716 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-tmp-dir" pod "9a0129d4-b54f-4081-9f28-3bc5c61dbf17" (UID: "9a0129d4-b54f-4081-9f28-3bc5c61dbf17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:17.526766 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.526753 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwfwb\" (UniqueName: \"kubernetes.io/projected/7c4a9040-3e1b-4c8c-b532-36140681da06-kube-api-access-wwfwb\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:17.526919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.526777 4335 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-model-cache\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:17.526919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.526792 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l25mc\" (UniqueName: \"kubernetes.io/projected/43393e2f-feb9-4698-8f80-5291391625b3-kube-api-access-l25mc\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:17.526919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.526805 4335 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4a9040-3e1b-4c8c-b532-36140681da06-tls-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:17.526919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.526819 4335 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4a9040-3e1b-4c8c-b532-36140681da06-dshm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:17.526919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.526833 4335 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43393e2f-feb9-4698-8f80-5291391625b3-tls-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:17.526919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.526844 4335 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43393e2f-feb9-4698-8f80-5291391625b3-dshm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:17.527132 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.527067 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-home" pod "9a0129d4-b54f-4081-9f28-3bc5c61dbf17" (UID: "9a0129d4-b54f-4081-9f28-3bc5c61dbf17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:17.528998 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.528971 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-kube-api-access-fgrtd" pod "9a0129d4-b54f-4081-9f28-3bc5c61dbf17" (UID: "9a0129d4-b54f-4081-9f28-3bc5c61dbf17"). InnerVolumeSpecName "kube-api-access-fgrtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:17.529349 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.529331 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-tls-certs" pod "9a0129d4-b54f-4081-9f28-3bc5c61dbf17" (UID: "9a0129d4-b54f-4081-9f28-3bc5c61dbf17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:17.529417 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.529360 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-dshm" pod "9a0129d4-b54f-4081-9f28-3bc5c61dbf17" (UID: "9a0129d4-b54f-4081-9f28-3bc5c61dbf17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:17.627946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.627874 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgrtd\" (UniqueName: \"kubernetes.io/projected/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-kube-api-access-fgrtd\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:17.627946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.627908 4335 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-tmp-dir\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:17.627946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.627922 4335 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-tls-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:17.627946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.627934 4335 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-home\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:17.627946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:17.627946 4335 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a0129d4-b54f-4081-9f28-3bc5c61dbf17-dshm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:18.037213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:18.037122 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" podUID="f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:37:18.342683 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:18.342652 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" Oct 30 22:37:18.342683 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:18.342662 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8" event={"ID":"43393e2f-feb9-4698-8f80-5291391625b3","Type":"ContainerDied","Data":"0b6f70bc2456cda11a204a54a962bfae85ff3f0c700ed8cbdf5d3725408141b7"} Oct 30 22:37:18.343203 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:18.342713 4335 scope.go:122] "RemoveContainer" containerID="2b2bee240b587aa4b71594af8427567f2e16e17d6315d1425b01bfa35d54f531" Oct 30 22:37:18.344361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:18.344343 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" Oct 30 22:37:18.344540 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:18.344395 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8" event={"ID":"7c4a9040-3e1b-4c8c-b532-36140681da06","Type":"ContainerDied","Data":"8a87cf84b734669937a2c0871905e441ee9dc06b27dbf4afc56b5c3527276f6a"} Oct 30 22:37:18.346000 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:18.345977 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" event={"ID":"9a0129d4-b54f-4081-9f28-3bc5c61dbf17","Type":"ContainerDied","Data":"2f089028b156b635c4a9871307b9acc4ea00efe07fc0027e406b2a12f2127d84"} Oct 30 22:37:18.346119 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:18.346064 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7" Oct 30 22:37:18.354748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:18.354727 4335 scope.go:122] "RemoveContainer" containerID="5df9f26475e5c5bc2e65c78008d1bdccfa3218ae11c8071770b1cb0b522ef702" Oct 30 22:37:18.364103 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:18.364087 4335 scope.go:122] "RemoveContainer" containerID="1ed59dcbc635e21756cadafbf5a84a3796ac53e87950eb92b137b3f7cda3682f" Oct 30 22:37:18.507275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:18.507251 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7"] Oct 30 22:37:18.556720 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:18.556687 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-536975/simulator-w4-536975-kserve-785485744-p27c7"] Oct 30 22:37:18.665311 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:18.665250 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8"] Oct 30 22:37:18.670213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:18.670190 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-536975/premium-w4-536975-kserve-844f65b8b9-c25r8"] Oct 30 22:37:18.779358 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:18.779336 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8"] Oct 30 22:37:18.785347 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:18.785323 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-536975/distinct-2-w4-536975-kserve-85946c9cf5-djhz8"] Oct 30 22:37:19.056634 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:19.056588 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43393e2f-feb9-4698-8f80-5291391625b3" path="/var/lib/kubelet/pods/43393e2f-feb9-4698-8f80-5291391625b3/volumes" Oct 30 22:37:19.057010 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:19.056995 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" path="/var/lib/kubelet/pods/7c4a9040-3e1b-4c8c-b532-36140681da06/volumes" Oct 30 22:37:19.057349 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:19.057337 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" path="/var/lib/kubelet/pods/9a0129d4-b54f-4081-9f28-3bc5c61dbf17/volumes" Oct 30 22:37:25.324940 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:37:25.324890 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:37:25.669096 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:25.669016 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:37:25.881323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:25.881290 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk"] Oct 30 22:37:25.930594 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:25.930537 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-536975-v4nmk"] Oct 30 22:37:27.055958 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:27.055925 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ada944-71f3-45f1-8311-70137f5a862b" path="/var/lib/kubelet/pods/71ada944-71f3-45f1-8311-70137f5a862b/volumes" Oct 30 22:37:28.037306 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:28.037270 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" podUID="f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:37:33.253465 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:33.253441 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" Oct 30 22:37:33.354692 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:33.354663 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31-plugins-config-volume\") pod \"f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31\" (UID: \"f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31\") " Oct 30 22:37:33.354854 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:33.354801 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31-kube-api-access-9ksg9\" (UniqueName: \"kubernetes.io/projected/f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31-kube-api-access-9ksg9\") pod \"f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31\" (UID: \"f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31\") " Oct 30 22:37:33.355065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:33.355042 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31-plugins-config-volume" pod "f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31" (UID: "f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:33.357044 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:33.357019 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31-kube-api-access-9ksg9" pod "f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31" (UID: "f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31"). InnerVolumeSpecName "kube-api-access-9ksg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:33.453785 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:33.453759 4335 generic.go:356] "Generic (PLEG): container finished" podID="f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31" containerID="cb07f312897257e9ba1b6d348af6b0c97bf28ba42d2441ae8711e0f747173e3e" exitCode=0 Oct 30 22:37:33.453891 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:33.453808 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" Oct 30 22:37:33.453891 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:33.453844 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" event={"ID":"f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31","Type":"ContainerDied","Data":"cb07f312897257e9ba1b6d348af6b0c97bf28ba42d2441ae8711e0f747173e3e"} Oct 30 22:37:33.453970 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:33.453900 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j" event={"ID":"f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31","Type":"ContainerDied","Data":"176449c40837e342582cfae257b96a013a6cede4b728cff8bb4ee870a4f5f352"} Oct 30 22:37:33.453970 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:33.453930 4335 scope.go:122] "RemoveContainer" containerID="cb07f312897257e9ba1b6d348af6b0c97bf28ba42d2441ae8711e0f747173e3e" Oct 30 22:37:33.455426 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:33.455404 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:33.455527 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:33.455429 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ksg9\" (UniqueName: \"kubernetes.io/projected/f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31-kube-api-access-9ksg9\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:33.463487 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:33.463472 4335 scope.go:122] "RemoveContainer" containerID="cb07f312897257e9ba1b6d348af6b0c97bf28ba42d2441ae8711e0f747173e3e" Oct 30 22:37:33.463870 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:37:33.463849 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb07f312897257e9ba1b6d348af6b0c97bf28ba42d2441ae8711e0f747173e3e\": container with ID starting with cb07f312897257e9ba1b6d348af6b0c97bf28ba42d2441ae8711e0f747173e3e not found: ID does not exist" containerID="cb07f312897257e9ba1b6d348af6b0c97bf28ba42d2441ae8711e0f747173e3e" Oct 30 22:37:33.463958 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:33.463876 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb07f312897257e9ba1b6d348af6b0c97bf28ba42d2441ae8711e0f747173e3e"} err="failed to get container status \"cb07f312897257e9ba1b6d348af6b0c97bf28ba42d2441ae8711e0f747173e3e\": rpc error: code = NotFound desc = could not find container \"cb07f312897257e9ba1b6d348af6b0c97bf28ba42d2441ae8711e0f747173e3e\": container with ID starting with cb07f312897257e9ba1b6d348af6b0c97bf28ba42d2441ae8711e0f747173e3e not found: ID does not exist" Oct 30 22:37:33.562733 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:33.562700 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4"] Oct 30 22:37:33.563039 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:33.563002 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" podUID="a3b3ae09-c03e-4c8a-95ea-31140419aa9a" containerName="istio-proxy" containerID="cri-o://6b103838b61676786b960148cbf1d5da2fead36d8e623199b8d032f1514e0f38" gracePeriod=30 Oct 30 22:37:33.627870 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:33.627809 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j"] Oct 30 22:37:33.636534 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:33.636501 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-536975-5b6494795c-zzr7j"] Oct 30 22:37:35.055737 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:35.055703 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31" path="/var/lib/kubelet/pods/f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31/volumes" Oct 30 22:37:37.216989 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:37.216955 4335 patch_prober.go:29] interesting pod/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:37:37.217378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:37.217010 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" podUID="a3b3ae09-c03e-4c8a-95ea-31140419aa9a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:37:38.821660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.821634 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:37:38.900070 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.900010 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-credential-socket\") pod \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " Oct 30 22:37:38.900070 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.900048 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-data\") pod \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " Oct 30 22:37:38.900262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.900086 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-workload-socket\") pod \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " Oct 30 22:37:38.900262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.900129 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-token\" (UniqueName: \"kubernetes.io/projected/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-token\") pod \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " Oct 30 22:37:38.900262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.900164 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-workload-certs\") pod \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " Oct 30 22:37:38.900436 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.900282 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-credential-socket" pod "a3b3ae09-c03e-4c8a-95ea-31140419aa9a" (UID: "a3b3ae09-c03e-4c8a-95ea-31140419aa9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:38.900436 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.900301 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-kube-api-access-cfx2k\" (UniqueName: \"kubernetes.io/projected/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-kube-api-access-cfx2k\") pod \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " Oct 30 22:37:38.900436 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.900366 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-envoy\") pod \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " Oct 30 22:37:38.900436 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.900383 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-workload-socket" pod "a3b3ae09-c03e-4c8a-95ea-31140419aa9a" (UID: "a3b3ae09-c03e-4c8a-95ea-31140419aa9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:38.900436 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.900424 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istiod-ca-cert\") pod \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " Oct 30 22:37:38.900780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.900448 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-data" pod "a3b3ae09-c03e-4c8a-95ea-31140419aa9a" (UID: "a3b3ae09-c03e-4c8a-95ea-31140419aa9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:38.900780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.900486 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-podinfo\") pod \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\" (UID: \"a3b3ae09-c03e-4c8a-95ea-31140419aa9a\") " Oct 30 22:37:38.900780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.900486 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-workload-certs" pod "a3b3ae09-c03e-4c8a-95ea-31140419aa9a" (UID: "a3b3ae09-c03e-4c8a-95ea-31140419aa9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:38.900967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.900839 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istiod-ca-cert" pod "a3b3ae09-c03e-4c8a-95ea-31140419aa9a" (UID: "a3b3ae09-c03e-4c8a-95ea-31140419aa9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:38.901149 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.901127 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:38.901213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.901157 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:38.901213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.901173 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:38.901213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.901189 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:38.901213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.901203 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:38.902545 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.902525 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-token" pod "a3b3ae09-c03e-4c8a-95ea-31140419aa9a" (UID: "a3b3ae09-c03e-4c8a-95ea-31140419aa9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:38.902811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.902792 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-podinfo" pod "a3b3ae09-c03e-4c8a-95ea-31140419aa9a" (UID: "a3b3ae09-c03e-4c8a-95ea-31140419aa9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:37:38.903073 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.903050 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-envoy" pod "a3b3ae09-c03e-4c8a-95ea-31140419aa9a" (UID: "a3b3ae09-c03e-4c8a-95ea-31140419aa9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:38.903170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:38.903075 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-kube-api-access-cfx2k" pod "a3b3ae09-c03e-4c8a-95ea-31140419aa9a" (UID: "a3b3ae09-c03e-4c8a-95ea-31140419aa9a"). InnerVolumeSpecName "kube-api-access-cfx2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:39.002328 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:39.002303 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:39.002328 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:39.002329 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:39.002464 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:39.002338 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfx2k\" (UniqueName: \"kubernetes.io/projected/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-kube-api-access-cfx2k\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:39.002464 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:39.002348 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3b3ae09-c03e-4c8a-95ea-31140419aa9a-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:39.491165 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:39.491133 4335 generic.go:356] "Generic (PLEG): container finished" podID="a3b3ae09-c03e-4c8a-95ea-31140419aa9a" containerID="6b103838b61676786b960148cbf1d5da2fead36d8e623199b8d032f1514e0f38" exitCode=0 Oct 30 22:37:39.491361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:39.491185 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" Oct 30 22:37:39.491361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:39.491229 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" event={"ID":"a3b3ae09-c03e-4c8a-95ea-31140419aa9a","Type":"ContainerDied","Data":"6b103838b61676786b960148cbf1d5da2fead36d8e623199b8d032f1514e0f38"} Oct 30 22:37:39.491361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:39.491294 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4" event={"ID":"a3b3ae09-c03e-4c8a-95ea-31140419aa9a","Type":"ContainerDied","Data":"40fcc84f4cb7578a687f914b465e0039f6da256ecd4906dcb09c825e97e0c188"} Oct 30 22:37:39.491361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:39.491326 4335 scope.go:122] "RemoveContainer" containerID="6b103838b61676786b960148cbf1d5da2fead36d8e623199b8d032f1514e0f38" Oct 30 22:37:39.500750 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:39.500732 4335 scope.go:122] "RemoveContainer" containerID="6b103838b61676786b960148cbf1d5da2fead36d8e623199b8d032f1514e0f38" Oct 30 22:37:39.501004 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:37:39.500986 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b103838b61676786b960148cbf1d5da2fead36d8e623199b8d032f1514e0f38\": container with ID starting with 6b103838b61676786b960148cbf1d5da2fead36d8e623199b8d032f1514e0f38 not found: ID does not exist" containerID="6b103838b61676786b960148cbf1d5da2fead36d8e623199b8d032f1514e0f38" Oct 30 22:37:39.501042 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:39.501013 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b103838b61676786b960148cbf1d5da2fead36d8e623199b8d032f1514e0f38"} err="failed to get container status \"6b103838b61676786b960148cbf1d5da2fead36d8e623199b8d032f1514e0f38\": rpc error: code = NotFound desc = could not find container \"6b103838b61676786b960148cbf1d5da2fead36d8e623199b8d032f1514e0f38\": container with ID starting with 6b103838b61676786b960148cbf1d5da2fead36d8e623199b8d032f1514e0f38 not found: ID does not exist" Oct 30 22:37:39.706017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:39.705986 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4"] Oct 30 22:37:39.755689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:39.755606 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-536975-openshift-default-5ddddcdd95-5mks4"] Oct 30 22:37:41.064816 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:41.064781 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3b3ae09-c03e-4c8a-95ea-31140419aa9a" path="/var/lib/kubelet/pods/a3b3ae09-c03e-4c8a-95ea-31140419aa9a/volumes" Oct 30 22:37:41.065221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:41.065109 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:37:48.887579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:48.887546 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bb56545f7-h6p47"] Oct 30 22:37:48.888087 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:48.887774 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7bb56545f7-h6p47" podUID="d25fe090-0a01-4e56-8337-f1edaa4ac618" containerName="manager" containerID="cri-o://2f3ac74c472a212a0b63e23960099bfd4a54c52a83a0cc05a023b354dc20e0d3" gracePeriod=30 Oct 30 22:37:49.134505 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.134482 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bb56545f7-h6p47" Oct 30 22:37:49.282136 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.282068 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d25fe090-0a01-4e56-8337-f1edaa4ac618-kube-api-access-9msh8\" (UniqueName: \"kubernetes.io/projected/d25fe090-0a01-4e56-8337-f1edaa4ac618-kube-api-access-9msh8\") pod \"d25fe090-0a01-4e56-8337-f1edaa4ac618\" (UID: \"d25fe090-0a01-4e56-8337-f1edaa4ac618\") " Oct 30 22:37:49.282293 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.282172 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d25fe090-0a01-4e56-8337-f1edaa4ac618-webhook-cert\" (UniqueName: \"kubernetes.io/secret/d25fe090-0a01-4e56-8337-f1edaa4ac618-webhook-cert\") pod \"d25fe090-0a01-4e56-8337-f1edaa4ac618\" (UID: \"d25fe090-0a01-4e56-8337-f1edaa4ac618\") " Oct 30 22:37:49.282293 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.282203 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d25fe090-0a01-4e56-8337-f1edaa4ac618-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25fe090-0a01-4e56-8337-f1edaa4ac618-metrics-certs\") pod \"d25fe090-0a01-4e56-8337-f1edaa4ac618\" (UID: \"d25fe090-0a01-4e56-8337-f1edaa4ac618\") " Oct 30 22:37:49.284377 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.284348 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25fe090-0a01-4e56-8337-f1edaa4ac618-kube-api-access-9msh8" pod "d25fe090-0a01-4e56-8337-f1edaa4ac618" (UID: "d25fe090-0a01-4e56-8337-f1edaa4ac618"). InnerVolumeSpecName "kube-api-access-9msh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:49.284511 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.284407 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d25fe090-0a01-4e56-8337-f1edaa4ac618-metrics-certs" pod "d25fe090-0a01-4e56-8337-f1edaa4ac618" (UID: "d25fe090-0a01-4e56-8337-f1edaa4ac618"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:49.284511 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.284433 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d25fe090-0a01-4e56-8337-f1edaa4ac618-webhook-cert" pod "d25fe090-0a01-4e56-8337-f1edaa4ac618" (UID: "d25fe090-0a01-4e56-8337-f1edaa4ac618"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:49.383266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.383239 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9msh8\" (UniqueName: \"kubernetes.io/projected/d25fe090-0a01-4e56-8337-f1edaa4ac618-kube-api-access-9msh8\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:49.383266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.383265 4335 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d25fe090-0a01-4e56-8337-f1edaa4ac618-webhook-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:49.383449 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.383275 4335 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25fe090-0a01-4e56-8337-f1edaa4ac618-metrics-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:49.564296 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.564262 4335 generic.go:356] "Generic (PLEG): container finished" podID="d25fe090-0a01-4e56-8337-f1edaa4ac618" containerID="2f3ac74c472a212a0b63e23960099bfd4a54c52a83a0cc05a023b354dc20e0d3" exitCode=0 Oct 30 22:37:49.564463 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.564375 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bb56545f7-h6p47" Oct 30 22:37:49.564463 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.564393 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bb56545f7-h6p47" event={"ID":"d25fe090-0a01-4e56-8337-f1edaa4ac618","Type":"ContainerDied","Data":"2f3ac74c472a212a0b63e23960099bfd4a54c52a83a0cc05a023b354dc20e0d3"} Oct 30 22:37:49.564463 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.564447 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bb56545f7-h6p47" event={"ID":"d25fe090-0a01-4e56-8337-f1edaa4ac618","Type":"ContainerDied","Data":"78bc19eb1cb491be01912940937342c2548cd5f9498e13557f90445e958b5ad4"} Oct 30 22:37:49.564680 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.564471 4335 scope.go:122] "RemoveContainer" containerID="2f3ac74c472a212a0b63e23960099bfd4a54c52a83a0cc05a023b354dc20e0d3" Oct 30 22:37:49.575102 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.575086 4335 scope.go:122] "RemoveContainer" containerID="2f3ac74c472a212a0b63e23960099bfd4a54c52a83a0cc05a023b354dc20e0d3" Oct 30 22:37:49.575407 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:37:49.575387 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3ac74c472a212a0b63e23960099bfd4a54c52a83a0cc05a023b354dc20e0d3\": container with ID starting with 2f3ac74c472a212a0b63e23960099bfd4a54c52a83a0cc05a023b354dc20e0d3 not found: ID does not exist" containerID="2f3ac74c472a212a0b63e23960099bfd4a54c52a83a0cc05a023b354dc20e0d3" Oct 30 22:37:49.575510 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.575418 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3ac74c472a212a0b63e23960099bfd4a54c52a83a0cc05a023b354dc20e0d3"} err="failed to get container status \"2f3ac74c472a212a0b63e23960099bfd4a54c52a83a0cc05a023b354dc20e0d3\": rpc error: code = NotFound desc = could not find container \"2f3ac74c472a212a0b63e23960099bfd4a54c52a83a0cc05a023b354dc20e0d3\": container with ID starting with 2f3ac74c472a212a0b63e23960099bfd4a54c52a83a0cc05a023b354dc20e0d3 not found: ID does not exist" Oct 30 22:37:49.732075 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.732047 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bb56545f7-h6p47"] Oct 30 22:37:49.779553 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:49.779527 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7bb56545f7-h6p47"] Oct 30 22:37:50.605147 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605117 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bb56545f7-hqm9d"] Oct 30 22:37:50.605533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605518 4335 cpu_manager.go:414] "containerMap: removing container" podUID="a3b3ae09-c03e-4c8a-95ea-31140419aa9a" containerName="istio-proxy" Oct 30 22:37:50.605592 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605537 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b3ae09-c03e-4c8a-95ea-31140419aa9a" containerName="istio-proxy" Oct 30 22:37:50.605592 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605550 4335 cpu_manager.go:414] "containerMap: removing container" podUID="65dd4857-0bb8-4daf-b312-2202c1b91795" containerName="revoke-keys" Oct 30 22:37:50.605592 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605555 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65dd4857-0bb8-4daf-b312-2202c1b91795" containerName="revoke-keys" Oct 30 22:37:50.605592 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605565 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31" containerName="payload-processing" Oct 30 22:37:50.605592 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605571 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31" containerName="payload-processing" Oct 30 22:37:50.605592 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605577 4335 cpu_manager.go:414] "containerMap: removing container" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" Oct 30 22:37:50.605592 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605581 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" Oct 30 22:37:50.605592 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605591 4335 cpu_manager.go:414] "containerMap: removing container" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" Oct 30 22:37:50.605879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605596 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" Oct 30 22:37:50.605879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605609 4335 cpu_manager.go:414] "containerMap: removing container" podUID="d25fe090-0a01-4e56-8337-f1edaa4ac618" containerName="manager" Oct 30 22:37:50.605879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605634 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25fe090-0a01-4e56-8337-f1edaa4ac618" containerName="manager" Oct 30 22:37:50.605879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605646 4335 cpu_manager.go:414] "containerMap: removing container" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" Oct 30 22:37:50.605879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605650 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" Oct 30 22:37:50.605879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605719 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25fe090-0a01-4e56-8337-f1edaa4ac618" containerName="manager" Oct 30 22:37:50.605879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605727 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="43393e2f-feb9-4698-8f80-5291391625b3" containerName="main" Oct 30 22:37:50.605879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605737 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b3ae09-c03e-4c8a-95ea-31140419aa9a" containerName="istio-proxy" Oct 30 22:37:50.605879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605742 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="65dd4857-0bb8-4daf-b312-2202c1b91795" containerName="revoke-keys" Oct 30 22:37:50.605879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605748 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f96f55b5-31e6-4b6a-9dc6-a6ef20a1be31" containerName="payload-processing" Oct 30 22:37:50.605879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605753 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0129d4-b54f-4081-9f28-3bc5c61dbf17" containerName="main" Oct 30 22:37:50.605879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.605758 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c4a9040-3e1b-4c8c-b532-36140681da06" containerName="main" Oct 30 22:37:50.610185 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.610170 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bb56545f7-hqm9d" Oct 30 22:37:50.662330 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.662305 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 22:37:50.693851 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.693802 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/daa0c1a3-e69c-4986-a664-8b6051a7c601-webhook-cert\") pod \"maas-controller-7bb56545f7-hqm9d\" (UID: \"daa0c1a3-e69c-4986-a664-8b6051a7c601\") " pod="opendatahub/maas-controller-7bb56545f7-hqm9d" Oct 30 22:37:50.693967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.693873 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa0c1a3-e69c-4986-a664-8b6051a7c601-metrics-certs\") pod \"maas-controller-7bb56545f7-hqm9d\" (UID: \"daa0c1a3-e69c-4986-a664-8b6051a7c601\") " pod="opendatahub/maas-controller-7bb56545f7-hqm9d" Oct 30 22:37:50.694050 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.693992 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvgvr\" (UniqueName: \"kubernetes.io/projected/daa0c1a3-e69c-4986-a664-8b6051a7c601-kube-api-access-cvgvr\") pod \"maas-controller-7bb56545f7-hqm9d\" (UID: \"daa0c1a3-e69c-4986-a664-8b6051a7c601\") " pod="opendatahub/maas-controller-7bb56545f7-hqm9d" Oct 30 22:37:50.711785 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.711765 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 22:37:50.711911 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.711896 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 22:37:50.711971 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.711908 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sr62x\"" Oct 30 22:37:50.715237 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.715213 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bb56545f7-hqm9d"] Oct 30 22:37:50.795414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.795389 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/daa0c1a3-e69c-4986-a664-8b6051a7c601-webhook-cert\") pod \"maas-controller-7bb56545f7-hqm9d\" (UID: \"daa0c1a3-e69c-4986-a664-8b6051a7c601\") " pod="opendatahub/maas-controller-7bb56545f7-hqm9d" Oct 30 22:37:50.795550 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.795424 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa0c1a3-e69c-4986-a664-8b6051a7c601-metrics-certs\") pod \"maas-controller-7bb56545f7-hqm9d\" (UID: \"daa0c1a3-e69c-4986-a664-8b6051a7c601\") " pod="opendatahub/maas-controller-7bb56545f7-hqm9d" Oct 30 22:37:50.795550 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.795477 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvgvr\" (UniqueName: \"kubernetes.io/projected/daa0c1a3-e69c-4986-a664-8b6051a7c601-kube-api-access-cvgvr\") pod \"maas-controller-7bb56545f7-hqm9d\" (UID: \"daa0c1a3-e69c-4986-a664-8b6051a7c601\") " pod="opendatahub/maas-controller-7bb56545f7-hqm9d" Oct 30 22:37:50.798213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.798191 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/daa0c1a3-e69c-4986-a664-8b6051a7c601-webhook-cert\") pod \"maas-controller-7bb56545f7-hqm9d\" (UID: \"daa0c1a3-e69c-4986-a664-8b6051a7c601\") " pod="opendatahub/maas-controller-7bb56545f7-hqm9d" Oct 30 22:37:50.798303 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.798191 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa0c1a3-e69c-4986-a664-8b6051a7c601-metrics-certs\") pod \"maas-controller-7bb56545f7-hqm9d\" (UID: \"daa0c1a3-e69c-4986-a664-8b6051a7c601\") " pod="opendatahub/maas-controller-7bb56545f7-hqm9d" Oct 30 22:37:50.857694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.857634 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvgvr\" (UniqueName: \"kubernetes.io/projected/daa0c1a3-e69c-4986-a664-8b6051a7c601-kube-api-access-cvgvr\") pod \"maas-controller-7bb56545f7-hqm9d\" (UID: \"daa0c1a3-e69c-4986-a664-8b6051a7c601\") " pod="opendatahub/maas-controller-7bb56545f7-hqm9d" Oct 30 22:37:50.923440 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:50.923418 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bb56545f7-hqm9d" Oct 30 22:37:51.056519 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:51.056487 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d25fe090-0a01-4e56-8337-f1edaa4ac618" path="/var/lib/kubelet/pods/d25fe090-0a01-4e56-8337-f1edaa4ac618/volumes" Oct 30 22:37:51.299149 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:51.299120 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bb56545f7-hqm9d"] Oct 30 22:37:51.305531 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:51.305505 4335 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:37:51.578060 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:51.578029 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bb56545f7-hqm9d" event={"ID":"daa0c1a3-e69c-4986-a664-8b6051a7c601","Type":"ContainerStarted","Data":"f2e936c2c8c382b7636bf1155b1196d7adcf4f4b32e43d19d04a36eff8998ec0"} Oct 30 22:37:52.585121 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:52.585091 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bb56545f7-hqm9d" event={"ID":"daa0c1a3-e69c-4986-a664-8b6051a7c601","Type":"ContainerStarted","Data":"0737872a394b800eaee1c144f2c2274976fdde6421fb823de012317c2c4bbbe4"} Oct 30 22:37:52.585493 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:52.585176 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bb56545f7-hqm9d" Oct 30 22:37:52.705524 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:37:52.705481 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7bb56545f7-hqm9d" podStartSLOduration=1.765568636 podStartE2EDuration="2.70546801s" totalImagesPullingTime="939.899374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:52.703323696 +0000 UTC m=+3205.803755447" watchObservedRunningTime="2026-10-30 22:37:52.70546801 +0000 UTC m=+3205.805899742" Oct 30 22:37:55.319590 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:37:55.319556 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:38:03.593236 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:03.593203 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bb56545f7-hqm9d" Oct 30 22:38:10.674223 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:10.674186 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9"] Oct 30 22:38:10.677498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:10.677483 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9" Oct 30 22:38:10.735725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:10.735705 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9"] Oct 30 22:38:10.781530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:10.781508 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:38:10.781694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:10.781644 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:38:10.781694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:10.781665 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:38:10.781694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:10.781669 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:38:10.867130 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:10.867103 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb65b2a7-047f-4a44-9498-124d804aad75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-ll4h9\" (UID: \"cb65b2a7-047f-4a44-9498-124d804aad75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9" Oct 30 22:38:10.867130 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:10.867135 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb65b2a7-047f-4a44-9498-124d804aad75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-ll4h9\" (UID: \"cb65b2a7-047f-4a44-9498-124d804aad75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9" Oct 30 22:38:10.867323 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:10.867175 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb677\" (UniqueName: \"kubernetes.io/projected/cb65b2a7-047f-4a44-9498-124d804aad75-kube-api-access-rb677\") pod \"maas-api-revoke-sub-85237e45-ll4h9\" (UID: \"cb65b2a7-047f-4a44-9498-124d804aad75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9" Oct 30 22:38:10.968514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:10.968446 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb65b2a7-047f-4a44-9498-124d804aad75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-ll4h9\" (UID: \"cb65b2a7-047f-4a44-9498-124d804aad75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9" Oct 30 22:38:10.968514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:10.968480 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb65b2a7-047f-4a44-9498-124d804aad75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-ll4h9\" (UID: \"cb65b2a7-047f-4a44-9498-124d804aad75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9" Oct 30 22:38:10.968727 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:10.968520 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb677\" (UniqueName: \"kubernetes.io/projected/cb65b2a7-047f-4a44-9498-124d804aad75-kube-api-access-rb677\") pod \"maas-api-revoke-sub-85237e45-ll4h9\" (UID: \"cb65b2a7-047f-4a44-9498-124d804aad75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9" Oct 30 22:38:10.969157 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:10.969135 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb65b2a7-047f-4a44-9498-124d804aad75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-ll4h9\" (UID: \"cb65b2a7-047f-4a44-9498-124d804aad75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9" Oct 30 22:38:10.971061 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:10.971034 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb65b2a7-047f-4a44-9498-124d804aad75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-ll4h9\" (UID: \"cb65b2a7-047f-4a44-9498-124d804aad75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9" Oct 30 22:38:11.071374 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:11.071345 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb677\" (UniqueName: \"kubernetes.io/projected/cb65b2a7-047f-4a44-9498-124d804aad75-kube-api-access-rb677\") pod \"maas-api-revoke-sub-85237e45-ll4h9\" (UID: \"cb65b2a7-047f-4a44-9498-124d804aad75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9" Oct 30 22:38:11.290206 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:11.290115 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9" Oct 30 22:38:11.665322 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:11.665296 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9"] Oct 30 22:38:11.666726 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:38:11.666698 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb65b2a7_047f_4a44_9498_124d804aad75.slice/crio-19626b3b12a8a0f3624fff2324bcda70f16c8a09683d77955bcfdd96d02b8dfe WatchSource:0}: Error finding container 19626b3b12a8a0f3624fff2324bcda70f16c8a09683d77955bcfdd96d02b8dfe: Status 404 returned error can't find the container with id 19626b3b12a8a0f3624fff2324bcda70f16c8a09683d77955bcfdd96d02b8dfe Oct 30 22:38:11.698739 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:11.698693 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9" event={"ID":"cb65b2a7-047f-4a44-9498-124d804aad75","Type":"ContainerStarted","Data":"19626b3b12a8a0f3624fff2324bcda70f16c8a09683d77955bcfdd96d02b8dfe"} Oct 30 22:38:12.706645 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:12.706594 4335 generic.go:356] "Generic (PLEG): container finished" podID="cb65b2a7-047f-4a44-9498-124d804aad75" containerID="cd082cec2f6d9dae3fbbab5f1228531384defdaffcc3b6dc431d036f3b3a7244" exitCode=0 Oct 30 22:38:12.706976 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:12.706644 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9" event={"ID":"cb65b2a7-047f-4a44-9498-124d804aad75","Type":"ContainerDied","Data":"cd082cec2f6d9dae3fbbab5f1228531384defdaffcc3b6dc431d036f3b3a7244"} Oct 30 22:38:13.843710 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:13.843026 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9" Oct 30 22:38:13.891315 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:13.891288 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb65b2a7-047f-4a44-9498-124d804aad75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb65b2a7-047f-4a44-9498-124d804aad75-maas-api-openshift-ca\") pod \"cb65b2a7-047f-4a44-9498-124d804aad75\" (UID: \"cb65b2a7-047f-4a44-9498-124d804aad75\") " Oct 30 22:38:13.891470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:13.891398 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb65b2a7-047f-4a44-9498-124d804aad75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb65b2a7-047f-4a44-9498-124d804aad75-maas-api-xks-ca\") pod \"cb65b2a7-047f-4a44-9498-124d804aad75\" (UID: \"cb65b2a7-047f-4a44-9498-124d804aad75\") " Oct 30 22:38:13.891470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:13.891444 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb65b2a7-047f-4a44-9498-124d804aad75-kube-api-access-rb677\" (UniqueName: \"kubernetes.io/projected/cb65b2a7-047f-4a44-9498-124d804aad75-kube-api-access-rb677\") pod \"cb65b2a7-047f-4a44-9498-124d804aad75\" (UID: \"cb65b2a7-047f-4a44-9498-124d804aad75\") " Oct 30 22:38:13.891610 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:13.891589 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb65b2a7-047f-4a44-9498-124d804aad75-maas-api-openshift-ca" pod "cb65b2a7-047f-4a44-9498-124d804aad75" (UID: "cb65b2a7-047f-4a44-9498-124d804aad75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:13.891726 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:13.891710 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb65b2a7-047f-4a44-9498-124d804aad75-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:13.893620 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:13.893588 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb65b2a7-047f-4a44-9498-124d804aad75-maas-api-xks-ca" pod "cb65b2a7-047f-4a44-9498-124d804aad75" (UID: "cb65b2a7-047f-4a44-9498-124d804aad75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:13.893691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:13.893604 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb65b2a7-047f-4a44-9498-124d804aad75-kube-api-access-rb677" pod "cb65b2a7-047f-4a44-9498-124d804aad75" (UID: "cb65b2a7-047f-4a44-9498-124d804aad75"). InnerVolumeSpecName "kube-api-access-rb677". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:13.992553 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:13.992498 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb65b2a7-047f-4a44-9498-124d804aad75-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:13.992553 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:13.992522 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb677\" (UniqueName: \"kubernetes.io/projected/cb65b2a7-047f-4a44-9498-124d804aad75-kube-api-access-rb677\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:14.720406 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:14.720366 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9" Oct 30 22:38:14.720406 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:14.720392 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9" event={"ID":"cb65b2a7-047f-4a44-9498-124d804aad75","Type":"ContainerDied","Data":"19626b3b12a8a0f3624fff2324bcda70f16c8a09683d77955bcfdd96d02b8dfe"} Oct 30 22:38:14.720720 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:14.720443 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19626b3b12a8a0f3624fff2324bcda70f16c8a09683d77955bcfdd96d02b8dfe" Oct 30 22:38:15.258691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.258655 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp"] Oct 30 22:38:15.259084 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.259043 4335 cpu_manager.go:414] "containerMap: removing container" podUID="cb65b2a7-047f-4a44-9498-124d804aad75" containerName="revoke-keys" Oct 30 22:38:15.259084 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.259056 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb65b2a7-047f-4a44-9498-124d804aad75" containerName="revoke-keys" Oct 30 22:38:15.259172 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.259135 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb65b2a7-047f-4a44-9498-124d804aad75" containerName="revoke-keys" Oct 30 22:38:15.262332 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.262311 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.303935 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.303908 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.304059 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.303940 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.304059 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.303980 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x5lq\" (UniqueName: \"kubernetes.io/projected/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-kube-api-access-6x5lq\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.304195 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.304105 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.304195 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.304142 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.304269 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.304202 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.304269 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.304236 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.304269 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.304256 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.304389 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.304274 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.367861 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.367836 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp"] Oct 30 22:38:15.405243 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.405214 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.405365 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.405248 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.405365 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.405264 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.405365 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.405286 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.405365 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.405322 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x5lq\" (UniqueName: \"kubernetes.io/projected/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-kube-api-access-6x5lq\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.405365 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.405349 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.405365 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.405365 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.405774 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.405388 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.405774 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.405438 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.405886 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.405836 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.405984 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.405896 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.405984 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.405966 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.406271 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.406088 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.406271 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.406168 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.407803 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.407781 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.407923 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.407907 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.409228 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.409192 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v5mg8\"" Oct 30 22:38:15.467078 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.467049 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.467207 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.467185 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x5lq\" (UniqueName: \"kubernetes.io/projected/181f13d9-d1ab-43a9-b6a0-7f4a72f7632b-kube-api-access-6x5lq\") pod \"maas-default-gateway-openshift-default-6cc458d887-7nqtp\" (UID: \"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:15.576042 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:15.576016 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:16.030537 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:38:16.030505 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181f13d9_d1ab_43a9_b6a0_7f4a72f7632b.slice/crio-2d15e714af64ec4e024c0eead9b65c0a8bb2e983f97b6ac9cb316b76b10d29f6 WatchSource:0}: Error finding container 2d15e714af64ec4e024c0eead9b65c0a8bb2e983f97b6ac9cb316b76b10d29f6: Status 404 returned error can't find the container with id 2d15e714af64ec4e024c0eead9b65c0a8bb2e983f97b6ac9cb316b76b10d29f6 Oct 30 22:38:16.031404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:16.031376 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp"] Oct 30 22:38:16.032695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:16.032657 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:38:16.032784 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:16.032728 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:38:16.032784 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:16.032757 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:38:16.735512 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:16.735473 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" event={"ID":"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b","Type":"ContainerStarted","Data":"fdac6315ae90a372198b14deeab465c679418348ac91786c4523f43a669e7193"} Oct 30 22:38:16.735998 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:16.735531 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" event={"ID":"181f13d9-d1ab-43a9-b6a0-7f4a72f7632b","Type":"ContainerStarted","Data":"2d15e714af64ec4e024c0eead9b65c0a8bb2e983f97b6ac9cb316b76b10d29f6"} Oct 30 22:38:16.858793 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:16.858724 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" podStartSLOduration=1.858708507 podStartE2EDuration="1.858708507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:16.855931314 +0000 UTC m=+3229.956363046" watchObservedRunningTime="2026-10-30 22:38:16.858708507 +0000 UTC m=+3229.959140336" Oct 30 22:38:17.576531 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:17.576496 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:17.577764 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:17.577740 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7nqtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Oct 30 22:38:17.577890 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:17.577796 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" podUID="181f13d9-d1ab-43a9-b6a0-7f4a72f7632b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Oct 30 22:38:18.576703 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:18.576649 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7nqtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Oct 30 22:38:18.577221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:18.576735 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" podUID="181f13d9-d1ab-43a9-b6a0-7f4a72f7632b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Oct 30 22:38:19.577235 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:19.577196 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7nqtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Oct 30 22:38:19.577773 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:19.577273 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" podUID="181f13d9-d1ab-43a9-b6a0-7f4a72f7632b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Oct 30 22:38:20.577318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:20.577284 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7nqtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Oct 30 22:38:20.577737 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:20.577338 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" podUID="181f13d9-d1ab-43a9-b6a0-7f4a72f7632b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Oct 30 22:38:21.577157 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:21.577075 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7nqtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Oct 30 22:38:21.577157 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:21.577132 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" podUID="181f13d9-d1ab-43a9-b6a0-7f4a72f7632b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Oct 30 22:38:22.576336 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:22.576298 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7nqtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Oct 30 22:38:22.576880 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:22.576350 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" podUID="181f13d9-d1ab-43a9-b6a0-7f4a72f7632b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Oct 30 22:38:24.186491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:24.186456 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:24.187219 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:24.187191 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:24.189102 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:24.189072 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" Oct 30 22:38:25.320312 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:38:25.320260 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:38:33.179427 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.179395 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bb56545f7-hqm9d"] Oct 30 22:38:33.179964 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.179657 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7bb56545f7-hqm9d" podUID="daa0c1a3-e69c-4986-a664-8b6051a7c601" containerName="manager" containerID="cri-o://0737872a394b800eaee1c144f2c2274976fdde6421fb823de012317c2c4bbbe4" gracePeriod=30 Oct 30 22:38:33.428571 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.428548 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bb56545f7-hqm9d" Oct 30 22:38:33.439651 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.439591 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daa0c1a3-e69c-4986-a664-8b6051a7c601-webhook-cert\" (UniqueName: \"kubernetes.io/secret/daa0c1a3-e69c-4986-a664-8b6051a7c601-webhook-cert\") pod \"daa0c1a3-e69c-4986-a664-8b6051a7c601\" (UID: \"daa0c1a3-e69c-4986-a664-8b6051a7c601\") " Oct 30 22:38:33.439731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.439676 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daa0c1a3-e69c-4986-a664-8b6051a7c601-metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa0c1a3-e69c-4986-a664-8b6051a7c601-metrics-certs\") pod \"daa0c1a3-e69c-4986-a664-8b6051a7c601\" (UID: \"daa0c1a3-e69c-4986-a664-8b6051a7c601\") " Oct 30 22:38:33.439731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.439710 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa0c1a3-e69c-4986-a664-8b6051a7c601-kube-api-access-cvgvr\" (UniqueName: \"kubernetes.io/projected/daa0c1a3-e69c-4986-a664-8b6051a7c601-kube-api-access-cvgvr\") pod \"daa0c1a3-e69c-4986-a664-8b6051a7c601\" (UID: \"daa0c1a3-e69c-4986-a664-8b6051a7c601\") " Oct 30 22:38:33.441731 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.441702 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa0c1a3-e69c-4986-a664-8b6051a7c601-webhook-cert" pod "daa0c1a3-e69c-4986-a664-8b6051a7c601" (UID: "daa0c1a3-e69c-4986-a664-8b6051a7c601"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:33.441822 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.441767 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa0c1a3-e69c-4986-a664-8b6051a7c601-kube-api-access-cvgvr" pod "daa0c1a3-e69c-4986-a664-8b6051a7c601" (UID: "daa0c1a3-e69c-4986-a664-8b6051a7c601"). InnerVolumeSpecName "kube-api-access-cvgvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:33.441822 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.441797 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa0c1a3-e69c-4986-a664-8b6051a7c601-metrics-certs" pod "daa0c1a3-e69c-4986-a664-8b6051a7c601" (UID: "daa0c1a3-e69c-4986-a664-8b6051a7c601"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:33.540518 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.540495 4335 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/daa0c1a3-e69c-4986-a664-8b6051a7c601-webhook-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:33.540518 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.540518 4335 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa0c1a3-e69c-4986-a664-8b6051a7c601-metrics-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:33.540670 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.540529 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvgvr\" (UniqueName: \"kubernetes.io/projected/daa0c1a3-e69c-4986-a664-8b6051a7c601-kube-api-access-cvgvr\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:33.843462 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.843428 4335 generic.go:356] "Generic (PLEG): container finished" podID="daa0c1a3-e69c-4986-a664-8b6051a7c601" containerID="0737872a394b800eaee1c144f2c2274976fdde6421fb823de012317c2c4bbbe4" exitCode=0 Oct 30 22:38:33.843658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.843470 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bb56545f7-hqm9d" Oct 30 22:38:33.843658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.843498 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bb56545f7-hqm9d" event={"ID":"daa0c1a3-e69c-4986-a664-8b6051a7c601","Type":"ContainerDied","Data":"0737872a394b800eaee1c144f2c2274976fdde6421fb823de012317c2c4bbbe4"} Oct 30 22:38:33.843658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.843540 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bb56545f7-hqm9d" event={"ID":"daa0c1a3-e69c-4986-a664-8b6051a7c601","Type":"ContainerDied","Data":"f2e936c2c8c382b7636bf1155b1196d7adcf4f4b32e43d19d04a36eff8998ec0"} Oct 30 22:38:33.843658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.843561 4335 scope.go:122] "RemoveContainer" containerID="0737872a394b800eaee1c144f2c2274976fdde6421fb823de012317c2c4bbbe4" Oct 30 22:38:33.853243 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.853226 4335 scope.go:122] "RemoveContainer" containerID="0737872a394b800eaee1c144f2c2274976fdde6421fb823de012317c2c4bbbe4" Oct 30 22:38:33.853490 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:38:33.853474 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0737872a394b800eaee1c144f2c2274976fdde6421fb823de012317c2c4bbbe4\": container with ID starting with 0737872a394b800eaee1c144f2c2274976fdde6421fb823de012317c2c4bbbe4 not found: ID does not exist" containerID="0737872a394b800eaee1c144f2c2274976fdde6421fb823de012317c2c4bbbe4" Oct 30 22:38:33.853558 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:33.853498 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0737872a394b800eaee1c144f2c2274976fdde6421fb823de012317c2c4bbbe4"} err="failed to get container status \"0737872a394b800eaee1c144f2c2274976fdde6421fb823de012317c2c4bbbe4\": rpc error: code = NotFound desc = could not find container \"0737872a394b800eaee1c144f2c2274976fdde6421fb823de012317c2c4bbbe4\": container with ID starting with 0737872a394b800eaee1c144f2c2274976fdde6421fb823de012317c2c4bbbe4 not found: ID does not exist" Oct 30 22:38:34.030303 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:34.030280 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bb56545f7-hqm9d"] Oct 30 22:38:34.079574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:34.079551 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7bb56545f7-hqm9d"] Oct 30 22:38:35.055939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.055910 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daa0c1a3-e69c-4986-a664-8b6051a7c601" path="/var/lib/kubelet/pods/daa0c1a3-e69c-4986-a664-8b6051a7c601/volumes" Oct 30 22:38:35.462676 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.462584 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bb56545f7-hrrcg"] Oct 30 22:38:35.463024 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.463010 4335 cpu_manager.go:414] "containerMap: removing container" podUID="daa0c1a3-e69c-4986-a664-8b6051a7c601" containerName="manager" Oct 30 22:38:35.463077 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.463026 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa0c1a3-e69c-4986-a664-8b6051a7c601" containerName="manager" Oct 30 22:38:35.463113 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.463093 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa0c1a3-e69c-4986-a664-8b6051a7c601" containerName="manager" Oct 30 22:38:35.467457 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.467436 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bb56545f7-hrrcg" Oct 30 22:38:35.557207 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.557178 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8299996-476b-4d62-bd03-a25a4a46645f-metrics-certs\") pod \"maas-controller-7bb56545f7-hrrcg\" (UID: \"a8299996-476b-4d62-bd03-a25a4a46645f\") " pod="opendatahub/maas-controller-7bb56545f7-hrrcg" Oct 30 22:38:35.557346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.557234 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdbdd\" (UniqueName: \"kubernetes.io/projected/a8299996-476b-4d62-bd03-a25a4a46645f-kube-api-access-rdbdd\") pod \"maas-controller-7bb56545f7-hrrcg\" (UID: \"a8299996-476b-4d62-bd03-a25a4a46645f\") " pod="opendatahub/maas-controller-7bb56545f7-hrrcg" Oct 30 22:38:35.557346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.557324 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a8299996-476b-4d62-bd03-a25a4a46645f-webhook-cert\") pod \"maas-controller-7bb56545f7-hrrcg\" (UID: \"a8299996-476b-4d62-bd03-a25a4a46645f\") " pod="opendatahub/maas-controller-7bb56545f7-hrrcg" Oct 30 22:38:35.569018 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.568985 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 22:38:35.569340 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.569046 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 22:38:35.569340 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.569092 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sr62x\"" Oct 30 22:38:35.569340 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.569116 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 22:38:35.572445 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.572423 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bb56545f7-hrrcg"] Oct 30 22:38:35.658272 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.658251 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8299996-476b-4d62-bd03-a25a4a46645f-metrics-certs\") pod \"maas-controller-7bb56545f7-hrrcg\" (UID: \"a8299996-476b-4d62-bd03-a25a4a46645f\") " pod="opendatahub/maas-controller-7bb56545f7-hrrcg" Oct 30 22:38:35.658392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.658303 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdbdd\" (UniqueName: \"kubernetes.io/projected/a8299996-476b-4d62-bd03-a25a4a46645f-kube-api-access-rdbdd\") pod \"maas-controller-7bb56545f7-hrrcg\" (UID: \"a8299996-476b-4d62-bd03-a25a4a46645f\") " pod="opendatahub/maas-controller-7bb56545f7-hrrcg" Oct 30 22:38:35.658392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.658336 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a8299996-476b-4d62-bd03-a25a4a46645f-webhook-cert\") pod \"maas-controller-7bb56545f7-hrrcg\" (UID: \"a8299996-476b-4d62-bd03-a25a4a46645f\") " pod="opendatahub/maas-controller-7bb56545f7-hrrcg" Oct 30 22:38:35.660902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.660883 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a8299996-476b-4d62-bd03-a25a4a46645f-webhook-cert\") pod \"maas-controller-7bb56545f7-hrrcg\" (UID: \"a8299996-476b-4d62-bd03-a25a4a46645f\") " pod="opendatahub/maas-controller-7bb56545f7-hrrcg" Oct 30 22:38:35.661042 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.661023 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8299996-476b-4d62-bd03-a25a4a46645f-metrics-certs\") pod \"maas-controller-7bb56545f7-hrrcg\" (UID: \"a8299996-476b-4d62-bd03-a25a4a46645f\") " pod="opendatahub/maas-controller-7bb56545f7-hrrcg" Oct 30 22:38:35.718775 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.718726 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdbdd\" (UniqueName: \"kubernetes.io/projected/a8299996-476b-4d62-bd03-a25a4a46645f-kube-api-access-rdbdd\") pod \"maas-controller-7bb56545f7-hrrcg\" (UID: \"a8299996-476b-4d62-bd03-a25a4a46645f\") " pod="opendatahub/maas-controller-7bb56545f7-hrrcg" Oct 30 22:38:35.780743 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:35.780724 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bb56545f7-hrrcg" Oct 30 22:38:36.160353 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:36.160328 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bb56545f7-hrrcg"] Oct 30 22:38:36.161712 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:38:36.161686 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8299996_476b_4d62_bd03_a25a4a46645f.slice/crio-39056e4eee3d13b14cd35c1f5ec12e8c9551ebcf1531c26e9a9ebc76ab94170c WatchSource:0}: Error finding container 39056e4eee3d13b14cd35c1f5ec12e8c9551ebcf1531c26e9a9ebc76ab94170c: Status 404 returned error can't find the container with id 39056e4eee3d13b14cd35c1f5ec12e8c9551ebcf1531c26e9a9ebc76ab94170c Oct 30 22:38:36.865630 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:36.865585 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bb56545f7-hrrcg" event={"ID":"a8299996-476b-4d62-bd03-a25a4a46645f","Type":"ContainerStarted","Data":"39056e4eee3d13b14cd35c1f5ec12e8c9551ebcf1531c26e9a9ebc76ab94170c"} Oct 30 22:38:37.872327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:37.872285 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bb56545f7-hrrcg" event={"ID":"a8299996-476b-4d62-bd03-a25a4a46645f","Type":"ContainerStarted","Data":"507d92b22ac1bb9cab1f4c056a35ccf7665bb8321f3e25f1b09ece5ec25ac4c7"} Oct 30 22:38:37.872816 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:37.872452 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bb56545f7-hrrcg" Oct 30 22:38:37.992479 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:37.992432 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7bb56545f7-hrrcg" podStartSLOduration=2.150930999 podStartE2EDuration="2.992420557s" totalImagesPullingTime="841.489558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:37.990070071 +0000 UTC m=+3251.090501803" watchObservedRunningTime="2026-10-30 22:38:37.992420557 +0000 UTC m=+3251.092852325" Oct 30 22:38:48.881668 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:38:48.881637 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bb56545f7-hrrcg" Oct 30 22:38:55.318906 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:38:55.318867 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:39:03.049733 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:03.049700 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8"] Oct 30 22:39:03.098527 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:03.098499 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee3bd233-c2xc8"] Oct 30 22:39:05.055340 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:05.055307 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20af7c9c-26cb-4ae1-b572-8a1ed8525b28" path="/var/lib/kubelet/pods/20af7c9c-26cb-4ae1-b572-8a1ed8525b28/volumes" Oct 30 22:39:24.362939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:24.362907 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:39:25.319751 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:39:25.319700 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:39:28.530304 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:28.530278 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:39:28.605234 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:28.605211 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:39:32.839673 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:32.839638 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd"] Oct 30 22:39:32.844419 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:32.844398 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd" Oct 30 22:39:32.899728 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:32.899703 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:39:32.899728 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:32.899714 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:39:32.948900 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:32.948875 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:39:32.949089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:32.949069 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:39:32.951643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:32.951625 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd"] Oct 30 22:39:33.006881 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:33.006857 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6711338e-7dff-4a43-b994-252eb34f8903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-q8wbd\" (UID: \"6711338e-7dff-4a43-b994-252eb34f8903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd" Oct 30 22:39:33.006975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:33.006912 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7j5w\" (UniqueName: \"kubernetes.io/projected/6711338e-7dff-4a43-b994-252eb34f8903-kube-api-access-n7j5w\") pod \"maas-api-revoke-sub-c1e271a9-q8wbd\" (UID: \"6711338e-7dff-4a43-b994-252eb34f8903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd" Oct 30 22:39:33.007022 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:33.007010 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6711338e-7dff-4a43-b994-252eb34f8903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-q8wbd\" (UID: \"6711338e-7dff-4a43-b994-252eb34f8903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd" Oct 30 22:39:33.107327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:33.107276 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6711338e-7dff-4a43-b994-252eb34f8903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-q8wbd\" (UID: \"6711338e-7dff-4a43-b994-252eb34f8903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd" Oct 30 22:39:33.107327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:33.107316 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6711338e-7dff-4a43-b994-252eb34f8903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-q8wbd\" (UID: \"6711338e-7dff-4a43-b994-252eb34f8903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd" Oct 30 22:39:33.107499 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:33.107358 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7j5w\" (UniqueName: \"kubernetes.io/projected/6711338e-7dff-4a43-b994-252eb34f8903-kube-api-access-n7j5w\") pod \"maas-api-revoke-sub-c1e271a9-q8wbd\" (UID: \"6711338e-7dff-4a43-b994-252eb34f8903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd" Oct 30 22:39:33.107986 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:33.107964 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6711338e-7dff-4a43-b994-252eb34f8903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-q8wbd\" (UID: \"6711338e-7dff-4a43-b994-252eb34f8903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd" Oct 30 22:39:33.109814 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:33.109796 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6711338e-7dff-4a43-b994-252eb34f8903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-q8wbd\" (UID: \"6711338e-7dff-4a43-b994-252eb34f8903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd" Oct 30 22:39:33.168488 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:33.168464 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7j5w\" (UniqueName: \"kubernetes.io/projected/6711338e-7dff-4a43-b994-252eb34f8903-kube-api-access-n7j5w\") pod \"maas-api-revoke-sub-c1e271a9-q8wbd\" (UID: \"6711338e-7dff-4a43-b994-252eb34f8903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd" Oct 30 22:39:33.457761 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:33.457701 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd" Oct 30 22:39:33.836394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:33.836363 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd"] Oct 30 22:39:33.837785 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:39:33.837759 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6711338e_7dff_4a43_b994_252eb34f8903.slice/crio-87dc005c8c1726b097c528a1b9c2d69900c46607436a861f2bc14c4aa0886896 WatchSource:0}: Error finding container 87dc005c8c1726b097c528a1b9c2d69900c46607436a861f2bc14c4aa0886896: Status 404 returned error can't find the container with id 87dc005c8c1726b097c528a1b9c2d69900c46607436a861f2bc14c4aa0886896 Oct 30 22:39:34.206713 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:34.206683 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd" event={"ID":"6711338e-7dff-4a43-b994-252eb34f8903","Type":"ContainerStarted","Data":"4b02e55fbce358cdcd6b778c18d8459a0a5ecc72a1c11cdec0f286ca28ca99b4"} Oct 30 22:39:34.207063 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:34.206740 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd" event={"ID":"6711338e-7dff-4a43-b994-252eb34f8903","Type":"ContainerStarted","Data":"87dc005c8c1726b097c528a1b9c2d69900c46607436a861f2bc14c4aa0886896"} Oct 30 22:39:34.325265 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:34.325222 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd" podStartSLOduration=2.325205652 podStartE2EDuration="2.325205652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:34.324464563 +0000 UTC m=+3307.424896296" watchObservedRunningTime="2026-10-30 22:39:34.325205652 +0000 UTC m=+3307.425637384" Oct 30 22:39:35.213598 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:35.213564 4335 generic.go:356] "Generic (PLEG): container finished" podID="6711338e-7dff-4a43-b994-252eb34f8903" containerID="4b02e55fbce358cdcd6b778c18d8459a0a5ecc72a1c11cdec0f286ca28ca99b4" exitCode=0 Oct 30 22:39:35.214016 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:35.213649 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd" event={"ID":"6711338e-7dff-4a43-b994-252eb34f8903","Type":"ContainerDied","Data":"4b02e55fbce358cdcd6b778c18d8459a0a5ecc72a1c11cdec0f286ca28ca99b4"} Oct 30 22:39:36.353767 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:36.353744 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd" Oct 30 22:39:36.537265 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:36.537205 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6711338e-7dff-4a43-b994-252eb34f8903-kube-api-access-n7j5w\" (UniqueName: \"kubernetes.io/projected/6711338e-7dff-4a43-b994-252eb34f8903-kube-api-access-n7j5w\") pod \"6711338e-7dff-4a43-b994-252eb34f8903\" (UID: \"6711338e-7dff-4a43-b994-252eb34f8903\") " Oct 30 22:39:36.537408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:36.537289 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6711338e-7dff-4a43-b994-252eb34f8903-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6711338e-7dff-4a43-b994-252eb34f8903-maas-api-xks-ca\") pod \"6711338e-7dff-4a43-b994-252eb34f8903\" (UID: \"6711338e-7dff-4a43-b994-252eb34f8903\") " Oct 30 22:39:36.537408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:36.537388 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6711338e-7dff-4a43-b994-252eb34f8903-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6711338e-7dff-4a43-b994-252eb34f8903-maas-api-openshift-ca\") pod \"6711338e-7dff-4a43-b994-252eb34f8903\" (UID: \"6711338e-7dff-4a43-b994-252eb34f8903\") " Oct 30 22:39:36.537734 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:36.537710 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6711338e-7dff-4a43-b994-252eb34f8903-maas-api-openshift-ca" pod "6711338e-7dff-4a43-b994-252eb34f8903" (UID: "6711338e-7dff-4a43-b994-252eb34f8903"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:39:36.539433 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:36.539413 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6711338e-7dff-4a43-b994-252eb34f8903-maas-api-xks-ca" pod "6711338e-7dff-4a43-b994-252eb34f8903" (UID: "6711338e-7dff-4a43-b994-252eb34f8903"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:39:36.539527 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:36.539512 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6711338e-7dff-4a43-b994-252eb34f8903-kube-api-access-n7j5w" pod "6711338e-7dff-4a43-b994-252eb34f8903" (UID: "6711338e-7dff-4a43-b994-252eb34f8903"). InnerVolumeSpecName "kube-api-access-n7j5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:36.638778 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:36.638754 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6711338e-7dff-4a43-b994-252eb34f8903-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:36.638778 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:36.638774 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6711338e-7dff-4a43-b994-252eb34f8903-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:36.638927 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:36.638784 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7j5w\" (UniqueName: \"kubernetes.io/projected/6711338e-7dff-4a43-b994-252eb34f8903-kube-api-access-n7j5w\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:39:37.230513 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:37.230486 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd" Oct 30 22:39:37.230646 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:37.230523 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd" event={"ID":"6711338e-7dff-4a43-b994-252eb34f8903","Type":"ContainerDied","Data":"87dc005c8c1726b097c528a1b9c2d69900c46607436a861f2bc14c4aa0886896"} Oct 30 22:39:37.230646 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:37.230566 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87dc005c8c1726b097c528a1b9c2d69900c46607436a861f2bc14c4aa0886896" Oct 30 22:39:45.640834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:45.640794 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:39:45.947361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:45.947288 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw"] Oct 30 22:39:45.947680 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:45.947667 4335 cpu_manager.go:414] "containerMap: removing container" podUID="6711338e-7dff-4a43-b994-252eb34f8903" containerName="revoke-keys" Oct 30 22:39:45.947741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:45.947682 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6711338e-7dff-4a43-b994-252eb34f8903" containerName="revoke-keys" Oct 30 22:39:45.947785 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:45.947767 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="6711338e-7dff-4a43-b994-252eb34f8903" containerName="revoke-keys" Oct 30 22:39:45.951113 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:45.951093 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.014229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.014198 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf808e7-af04-425e-88db-2ebeb7e44240-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.014229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.014233 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf808e7-af04-425e-88db-2ebeb7e44240-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.014426 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.014252 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf808e7-af04-425e-88db-2ebeb7e44240-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.014426 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.014385 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf808e7-af04-425e-88db-2ebeb7e44240-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.014499 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.014431 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xjmc\" (UniqueName: \"kubernetes.io/projected/bbf808e7-af04-425e-88db-2ebeb7e44240-kube-api-access-2xjmc\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.014499 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.014451 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf808e7-af04-425e-88db-2ebeb7e44240-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.014499 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.014473 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf808e7-af04-425e-88db-2ebeb7e44240-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.014601 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.014501 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf808e7-af04-425e-88db-2ebeb7e44240-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.014601 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.014572 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf808e7-af04-425e-88db-2ebeb7e44240-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.057652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.057608 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw"] Oct 30 22:39:46.115069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.115034 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf808e7-af04-425e-88db-2ebeb7e44240-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.115069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.115067 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf808e7-af04-425e-88db-2ebeb7e44240-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.115327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.115083 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf808e7-af04-425e-88db-2ebeb7e44240-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.115327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.115108 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf808e7-af04-425e-88db-2ebeb7e44240-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.115327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.115129 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjmc\" (UniqueName: \"kubernetes.io/projected/bbf808e7-af04-425e-88db-2ebeb7e44240-kube-api-access-2xjmc\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.115327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.115157 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf808e7-af04-425e-88db-2ebeb7e44240-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.115327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.115186 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf808e7-af04-425e-88db-2ebeb7e44240-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.115327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.115227 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf808e7-af04-425e-88db-2ebeb7e44240-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.115327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.115269 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf808e7-af04-425e-88db-2ebeb7e44240-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.115717 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.115575 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf808e7-af04-425e-88db-2ebeb7e44240-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.115717 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.115667 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf808e7-af04-425e-88db-2ebeb7e44240-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.115943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.115916 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf808e7-af04-425e-88db-2ebeb7e44240-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.116193 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.115945 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf808e7-af04-425e-88db-2ebeb7e44240-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.116299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.116269 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf808e7-af04-425e-88db-2ebeb7e44240-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.117635 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.117597 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf808e7-af04-425e-88db-2ebeb7e44240-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.117862 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.117846 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf808e7-af04-425e-88db-2ebeb7e44240-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.180088 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.180066 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xjmc\" (UniqueName: \"kubernetes.io/projected/bbf808e7-af04-425e-88db-2ebeb7e44240-kube-api-access-2xjmc\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.180269 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.180249 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf808e7-af04-425e-88db-2ebeb7e44240-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-jr7cw\" (UID: \"bbf808e7-af04-425e-88db-2ebeb7e44240\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.264652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.264597 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:46.703515 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:39:46.703478 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf808e7_af04_425e_88db_2ebeb7e44240.slice/crio-81b68a59c4e7b59e80d5b35b200a98e4a96836fb14f3be578f797a4bec875231 WatchSource:0}: Error finding container 81b68a59c4e7b59e80d5b35b200a98e4a96836fb14f3be578f797a4bec875231: Status 404 returned error can't find the container with id 81b68a59c4e7b59e80d5b35b200a98e4a96836fb14f3be578f797a4bec875231 Oct 30 22:39:46.704334 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.704309 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw"] Oct 30 22:39:46.705817 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.705784 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:39:46.705924 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.705858 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:39:46.705924 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:46.705903 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:39:47.323656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:47.318657 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" event={"ID":"bbf808e7-af04-425e-88db-2ebeb7e44240","Type":"ContainerStarted","Data":"c5e78cef01383f1cbfb85002b61879bfcf55cd8e41f68a47940b84558b5b2dc1"} Oct 30 22:39:47.323656 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:47.318779 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" event={"ID":"bbf808e7-af04-425e-88db-2ebeb7e44240","Type":"ContainerStarted","Data":"81b68a59c4e7b59e80d5b35b200a98e4a96836fb14f3be578f797a4bec875231"} Oct 30 22:39:47.451990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:47.451928 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" podStartSLOduration=2.451909094 podStartE2EDuration="2.451909094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:47.450697329 +0000 UTC m=+3320.551129061" watchObservedRunningTime="2026-10-30 22:39:47.451909094 +0000 UTC m=+3320.552340827" Oct 30 22:39:48.264834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:48.264801 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:48.266292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:48.266266 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jr7cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body= Oct 30 22:39:48.266433 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:48.266327 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" podUID="bbf808e7-af04-425e-88db-2ebeb7e44240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" Oct 30 22:39:49.265937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:49.265903 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jr7cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body= Oct 30 22:39:49.266324 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:49.265972 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" podUID="bbf808e7-af04-425e-88db-2ebeb7e44240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" Oct 30 22:39:50.265173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:50.265131 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jr7cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body= Oct 30 22:39:50.265394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:50.265184 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" podUID="bbf808e7-af04-425e-88db-2ebeb7e44240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" Oct 30 22:39:51.265976 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:51.265941 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jr7cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body= Oct 30 22:39:51.266446 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:51.266009 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" podUID="bbf808e7-af04-425e-88db-2ebeb7e44240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" Oct 30 22:39:52.265920 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:52.265878 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jr7cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body= Oct 30 22:39:52.266183 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:52.265948 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" podUID="bbf808e7-af04-425e-88db-2ebeb7e44240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" Oct 30 22:39:53.265779 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:53.265738 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jr7cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body= Oct 30 22:39:53.266103 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:53.265788 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" podUID="bbf808e7-af04-425e-88db-2ebeb7e44240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" Oct 30 22:39:55.265405 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:55.265372 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jr7cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:39:55.265953 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:55.265420 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" podUID="bbf808e7-af04-425e-88db-2ebeb7e44240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:39:55.319497 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:39:55.319458 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:39:56.306801 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:56.306752 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jr7cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:39:56.307315 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:56.306813 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" podUID="bbf808e7-af04-425e-88db-2ebeb7e44240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:39:56.825342 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:56.825306 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:39:57.113625 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:57.113537 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:57.113836 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:57.113817 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:39:57.114487 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:39:57.114470 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" Oct 30 22:40:01.076822 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.076788 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw"] Oct 30 22:40:01.080368 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.080352 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.167429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.167396 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.167555 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.167541 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.167654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.167589 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.167733 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.167693 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.167801 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.167758 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.167868 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.167796 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.167868 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.167848 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.167977 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.167876 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.167977 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.167910 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4ht8\" (UniqueName: \"kubernetes.io/projected/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-kube-api-access-v4ht8\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.269128 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.269098 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.269241 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.269148 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.269241 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.269190 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.269241 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.269215 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.269452 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.269255 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.269452 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.269405 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.269600 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.269475 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.269600 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.269544 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.269600 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.269571 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.269600 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.269592 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4ht8\" (UniqueName: \"kubernetes.io/projected/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-kube-api-access-v4ht8\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.269865 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.269648 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.269865 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.269714 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.269995 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.269864 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.270104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.270085 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.271698 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.271674 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.272047 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.272027 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.308903 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.308878 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw"] Oct 30 22:40:01.335495 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.335441 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4ht8\" (UniqueName: \"kubernetes.io/projected/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-kube-api-access-v4ht8\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.345156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.345133 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed2ae60-d2b4-4ce7-b48b-295a7eabe663-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-mjgkw\" (UID: \"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.393988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.393967 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:01.823694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.823671 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw"] Oct 30 22:40:01.824523 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:40:01.824500 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed2ae60_d2b4_4ce7_b48b_295a7eabe663.slice/crio-29219099be016cb733d266ac4fb9c46e3a3efcbabd845081cc76943d8b38e70d WatchSource:0}: Error finding container 29219099be016cb733d266ac4fb9c46e3a3efcbabd845081cc76943d8b38e70d: Status 404 returned error can't find the container with id 29219099be016cb733d266ac4fb9c46e3a3efcbabd845081cc76943d8b38e70d Oct 30 22:40:01.826531 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.826500 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:40:01.826585 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.826564 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:40:01.826653 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:01.826594 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:40:02.424713 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:02.424604 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" event={"ID":"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663","Type":"ContainerStarted","Data":"7da51b794a86ac8c717c9f9d8ad67824d21ec957b895ae850d74f5dd0f8e9888"} Oct 30 22:40:02.424713 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:02.424732 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" event={"ID":"4ed2ae60-d2b4-4ce7-b48b-295a7eabe663","Type":"ContainerStarted","Data":"29219099be016cb733d266ac4fb9c46e3a3efcbabd845081cc76943d8b38e70d"} Oct 30 22:40:02.547722 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:02.547671 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" podStartSLOduration=1.547655635 podStartE2EDuration="1.547655635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:02.546406737 +0000 UTC m=+3335.646838470" watchObservedRunningTime="2026-10-30 22:40:02.547655635 +0000 UTC m=+3335.648087367" Oct 30 22:40:03.394516 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:03.394480 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:03.401639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:03.399718 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-mjgkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Oct 30 22:40:03.401639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:03.399790 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" podUID="4ed2ae60-d2b4-4ce7-b48b-295a7eabe663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Oct 30 22:40:04.167163 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.167134 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2"] Oct 30 22:40:04.170933 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.170912 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2" Oct 30 22:40:04.225063 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.225007 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:40:04.225063 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.225061 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:40:04.225287 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.225261 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:40:04.225346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.225312 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:40:04.229410 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.229387 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2"] Oct 30 22:40:04.321525 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.321495 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85124c4-eefc-45c3-8a65-bb4b2464446d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-trtl2\" (UID: \"a85124c4-eefc-45c3-8a65-bb4b2464446d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2" Oct 30 22:40:04.321711 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.321543 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvlcs\" (UniqueName: \"kubernetes.io/projected/a85124c4-eefc-45c3-8a65-bb4b2464446d-kube-api-access-dvlcs\") pod \"maas-api-revoke-sub-2e1f5b5f-trtl2\" (UID: \"a85124c4-eefc-45c3-8a65-bb4b2464446d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2" Oct 30 22:40:04.321711 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.321599 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85124c4-eefc-45c3-8a65-bb4b2464446d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-trtl2\" (UID: \"a85124c4-eefc-45c3-8a65-bb4b2464446d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2" Oct 30 22:40:04.394734 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.394706 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-mjgkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Oct 30 22:40:04.394878 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.394748 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" podUID="4ed2ae60-d2b4-4ce7-b48b-295a7eabe663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Oct 30 22:40:04.422233 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.422163 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85124c4-eefc-45c3-8a65-bb4b2464446d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-trtl2\" (UID: \"a85124c4-eefc-45c3-8a65-bb4b2464446d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2" Oct 30 22:40:04.422233 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.422227 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85124c4-eefc-45c3-8a65-bb4b2464446d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-trtl2\" (UID: \"a85124c4-eefc-45c3-8a65-bb4b2464446d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2" Oct 30 22:40:04.422438 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.422257 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvlcs\" (UniqueName: \"kubernetes.io/projected/a85124c4-eefc-45c3-8a65-bb4b2464446d-kube-api-access-dvlcs\") pod \"maas-api-revoke-sub-2e1f5b5f-trtl2\" (UID: \"a85124c4-eefc-45c3-8a65-bb4b2464446d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2" Oct 30 22:40:04.423042 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.423018 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85124c4-eefc-45c3-8a65-bb4b2464446d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-trtl2\" (UID: \"a85124c4-eefc-45c3-8a65-bb4b2464446d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2" Oct 30 22:40:04.424942 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.424921 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85124c4-eefc-45c3-8a65-bb4b2464446d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-trtl2\" (UID: \"a85124c4-eefc-45c3-8a65-bb4b2464446d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2" Oct 30 22:40:04.483318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.483294 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvlcs\" (UniqueName: \"kubernetes.io/projected/a85124c4-eefc-45c3-8a65-bb4b2464446d-kube-api-access-dvlcs\") pod \"maas-api-revoke-sub-2e1f5b5f-trtl2\" (UID: \"a85124c4-eefc-45c3-8a65-bb4b2464446d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2" Oct 30 22:40:04.484901 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.484878 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2" Oct 30 22:40:04.891887 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:04.891855 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2"] Oct 30 22:40:04.894458 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:40:04.894423 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85124c4_eefc_45c3_8a65_bb4b2464446d.slice/crio-35ce1100fba19cd7188d13e9c4624f3c4c5a33709e0321914d6247d6dff5d926 WatchSource:0}: Error finding container 35ce1100fba19cd7188d13e9c4624f3c4c5a33709e0321914d6247d6dff5d926: Status 404 returned error can't find the container with id 35ce1100fba19cd7188d13e9c4624f3c4c5a33709e0321914d6247d6dff5d926 Oct 30 22:40:05.394929 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:05.394899 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-mjgkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Oct 30 22:40:05.395297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:05.394948 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" podUID="4ed2ae60-d2b4-4ce7-b48b-295a7eabe663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Oct 30 22:40:05.445760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:05.445494 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2" event={"ID":"a85124c4-eefc-45c3-8a65-bb4b2464446d","Type":"ContainerStarted","Data":"ea680fa5b775d77a837bccfef06c8fc9a81603e085008ee4e4789dbd2d324e70"} Oct 30 22:40:05.445760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:05.445547 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2" event={"ID":"a85124c4-eefc-45c3-8a65-bb4b2464446d","Type":"ContainerStarted","Data":"35ce1100fba19cd7188d13e9c4624f3c4c5a33709e0321914d6247d6dff5d926"} Oct 30 22:40:05.578341 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:05.578289 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2" podStartSLOduration=1.5782746460000001 podStartE2EDuration="1.578274646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:05.576979472 +0000 UTC m=+3338.677411205" watchObservedRunningTime="2026-10-30 22:40:05.578274646 +0000 UTC m=+3338.678706378" Oct 30 22:40:06.394386 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:06.394355 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-mjgkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Oct 30 22:40:06.394575 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:06.394405 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" podUID="4ed2ae60-d2b4-4ce7-b48b-295a7eabe663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Oct 30 22:40:06.452885 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:06.452821 4335 generic.go:356] "Generic (PLEG): container finished" podID="a85124c4-eefc-45c3-8a65-bb4b2464446d" containerID="ea680fa5b775d77a837bccfef06c8fc9a81603e085008ee4e4789dbd2d324e70" exitCode=0 Oct 30 22:40:06.453257 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:06.452940 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2" event={"ID":"a85124c4-eefc-45c3-8a65-bb4b2464446d","Type":"ContainerDied","Data":"ea680fa5b775d77a837bccfef06c8fc9a81603e085008ee4e4789dbd2d324e70"} Oct 30 22:40:07.394985 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:07.394950 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-mjgkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Oct 30 22:40:07.395175 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:07.395039 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" podUID="4ed2ae60-d2b4-4ce7-b48b-295a7eabe663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Oct 30 22:40:07.628248 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:07.628223 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2" Oct 30 22:40:07.756866 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:07.756776 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85124c4-eefc-45c3-8a65-bb4b2464446d-kube-api-access-dvlcs\" (UniqueName: \"kubernetes.io/projected/a85124c4-eefc-45c3-8a65-bb4b2464446d-kube-api-access-dvlcs\") pod \"a85124c4-eefc-45c3-8a65-bb4b2464446d\" (UID: \"a85124c4-eefc-45c3-8a65-bb4b2464446d\") " Oct 30 22:40:07.757035 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:07.756907 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a85124c4-eefc-45c3-8a65-bb4b2464446d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85124c4-eefc-45c3-8a65-bb4b2464446d-maas-api-openshift-ca\") pod \"a85124c4-eefc-45c3-8a65-bb4b2464446d\" (UID: \"a85124c4-eefc-45c3-8a65-bb4b2464446d\") " Oct 30 22:40:07.757035 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:07.757012 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a85124c4-eefc-45c3-8a65-bb4b2464446d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85124c4-eefc-45c3-8a65-bb4b2464446d-maas-api-xks-ca\") pod \"a85124c4-eefc-45c3-8a65-bb4b2464446d\" (UID: \"a85124c4-eefc-45c3-8a65-bb4b2464446d\") " Oct 30 22:40:07.757300 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:07.757279 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85124c4-eefc-45c3-8a65-bb4b2464446d-maas-api-openshift-ca" pod "a85124c4-eefc-45c3-8a65-bb4b2464446d" (UID: "a85124c4-eefc-45c3-8a65-bb4b2464446d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:07.757460 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:07.757444 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85124c4-eefc-45c3-8a65-bb4b2464446d-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:07.759124 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:07.759093 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85124c4-eefc-45c3-8a65-bb4b2464446d-kube-api-access-dvlcs" pod "a85124c4-eefc-45c3-8a65-bb4b2464446d" (UID: "a85124c4-eefc-45c3-8a65-bb4b2464446d"). InnerVolumeSpecName "kube-api-access-dvlcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:07.759683 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:07.759649 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85124c4-eefc-45c3-8a65-bb4b2464446d-maas-api-xks-ca" pod "a85124c4-eefc-45c3-8a65-bb4b2464446d" (UID: "a85124c4-eefc-45c3-8a65-bb4b2464446d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:07.858536 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:07.858510 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvlcs\" (UniqueName: \"kubernetes.io/projected/a85124c4-eefc-45c3-8a65-bb4b2464446d-kube-api-access-dvlcs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:07.858536 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:07.858534 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85124c4-eefc-45c3-8a65-bb4b2464446d-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:08.395154 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:08.395127 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-mjgkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Oct 30 22:40:08.395291 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:08.395172 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" podUID="4ed2ae60-d2b4-4ce7-b48b-295a7eabe663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Oct 30 22:40:08.466700 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:08.466678 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2" Oct 30 22:40:08.466824 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:08.466717 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2" event={"ID":"a85124c4-eefc-45c3-8a65-bb4b2464446d","Type":"ContainerDied","Data":"35ce1100fba19cd7188d13e9c4624f3c4c5a33709e0321914d6247d6dff5d926"} Oct 30 22:40:08.466824 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:08.466763 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35ce1100fba19cd7188d13e9c4624f3c4c5a33709e0321914d6247d6dff5d926" Oct 30 22:40:10.002357 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:10.002320 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:10.002899 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:10.002821 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:10.003652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:10.003606 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" Oct 30 22:40:16.617750 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.617713 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292"] Oct 30 22:40:16.618366 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.618343 4335 cpu_manager.go:414] "containerMap: removing container" podUID="a85124c4-eefc-45c3-8a65-bb4b2464446d" containerName="revoke-keys" Oct 30 22:40:16.618421 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.618372 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85124c4-eefc-45c3-8a65-bb4b2464446d" containerName="revoke-keys" Oct 30 22:40:16.618520 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.618507 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85124c4-eefc-45c3-8a65-bb4b2464446d" containerName="revoke-keys" Oct 30 22:40:16.632109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.632079 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292" Oct 30 22:40:16.678122 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.678090 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292"] Oct 30 22:40:16.723923 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.723892 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:40:16.723923 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.723916 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:40:16.724118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.723925 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:40:16.734086 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.734065 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:40:16.735470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.735452 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hwg5\" (UniqueName: \"kubernetes.io/projected/9669d8a2-2429-43a7-820e-2330ed2dbfcd-kube-api-access-7hwg5\") pod \"maas-api-revoke-sub-ec843018-vl292\" (UID: \"9669d8a2-2429-43a7-820e-2330ed2dbfcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292" Oct 30 22:40:16.735541 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.735515 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9669d8a2-2429-43a7-820e-2330ed2dbfcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-vl292\" (UID: \"9669d8a2-2429-43a7-820e-2330ed2dbfcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292" Oct 30 22:40:16.735670 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.735657 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9669d8a2-2429-43a7-820e-2330ed2dbfcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-vl292\" (UID: \"9669d8a2-2429-43a7-820e-2330ed2dbfcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292" Oct 30 22:40:16.836099 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.836070 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hwg5\" (UniqueName: \"kubernetes.io/projected/9669d8a2-2429-43a7-820e-2330ed2dbfcd-kube-api-access-7hwg5\") pod \"maas-api-revoke-sub-ec843018-vl292\" (UID: \"9669d8a2-2429-43a7-820e-2330ed2dbfcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292" Oct 30 22:40:16.836232 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.836140 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9669d8a2-2429-43a7-820e-2330ed2dbfcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-vl292\" (UID: \"9669d8a2-2429-43a7-820e-2330ed2dbfcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292" Oct 30 22:40:16.836232 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.836167 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9669d8a2-2429-43a7-820e-2330ed2dbfcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-vl292\" (UID: \"9669d8a2-2429-43a7-820e-2330ed2dbfcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292" Oct 30 22:40:16.836866 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.836837 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9669d8a2-2429-43a7-820e-2330ed2dbfcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-vl292\" (UID: \"9669d8a2-2429-43a7-820e-2330ed2dbfcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292" Oct 30 22:40:16.838751 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.838726 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9669d8a2-2429-43a7-820e-2330ed2dbfcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-vl292\" (UID: \"9669d8a2-2429-43a7-820e-2330ed2dbfcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292" Oct 30 22:40:16.897118 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.897064 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hwg5\" (UniqueName: \"kubernetes.io/projected/9669d8a2-2429-43a7-820e-2330ed2dbfcd-kube-api-access-7hwg5\") pod \"maas-api-revoke-sub-ec843018-vl292\" (UID: \"9669d8a2-2429-43a7-820e-2330ed2dbfcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292" Oct 30 22:40:16.938896 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.938875 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:40:16.947460 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:16.947440 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292" Oct 30 22:40:17.324304 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:17.324264 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292"] Oct 30 22:40:17.325877 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:40:17.325846 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9669d8a2_2429_43a7_820e_2330ed2dbfcd.slice/crio-4b3b505edf87476037e46db179df051499b0b76e5f6309aae894d302cd2be4b6 WatchSource:0}: Error finding container 4b3b505edf87476037e46db179df051499b0b76e5f6309aae894d302cd2be4b6: Status 404 returned error can't find the container with id 4b3b505edf87476037e46db179df051499b0b76e5f6309aae894d302cd2be4b6 Oct 30 22:40:17.528922 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:17.528896 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292" event={"ID":"9669d8a2-2429-43a7-820e-2330ed2dbfcd","Type":"ContainerStarted","Data":"4b3b505edf87476037e46db179df051499b0b76e5f6309aae894d302cd2be4b6"} Oct 30 22:40:18.540842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:18.540746 4335 generic.go:356] "Generic (PLEG): container finished" podID="9669d8a2-2429-43a7-820e-2330ed2dbfcd" containerID="c84470a93c55c8e2fae45dabe3a001f0109005f94499480a47d1f2eebde7c660" exitCode=0 Oct 30 22:40:18.540842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:18.540815 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292" event={"ID":"9669d8a2-2429-43a7-820e-2330ed2dbfcd","Type":"ContainerDied","Data":"c84470a93c55c8e2fae45dabe3a001f0109005f94499480a47d1f2eebde7c660"} Oct 30 22:40:19.713077 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:19.712989 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292" Oct 30 22:40:19.777596 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:19.777570 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9669d8a2-2429-43a7-820e-2330ed2dbfcd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9669d8a2-2429-43a7-820e-2330ed2dbfcd-maas-api-xks-ca\") pod \"9669d8a2-2429-43a7-820e-2330ed2dbfcd\" (UID: \"9669d8a2-2429-43a7-820e-2330ed2dbfcd\") " Oct 30 22:40:19.777722 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:19.777669 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9669d8a2-2429-43a7-820e-2330ed2dbfcd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9669d8a2-2429-43a7-820e-2330ed2dbfcd-maas-api-openshift-ca\") pod \"9669d8a2-2429-43a7-820e-2330ed2dbfcd\" (UID: \"9669d8a2-2429-43a7-820e-2330ed2dbfcd\") " Oct 30 22:40:19.777772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:19.777721 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9669d8a2-2429-43a7-820e-2330ed2dbfcd-kube-api-access-7hwg5\" (UniqueName: \"kubernetes.io/projected/9669d8a2-2429-43a7-820e-2330ed2dbfcd-kube-api-access-7hwg5\") pod \"9669d8a2-2429-43a7-820e-2330ed2dbfcd\" (UID: \"9669d8a2-2429-43a7-820e-2330ed2dbfcd\") " Oct 30 22:40:19.778034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:19.778005 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9669d8a2-2429-43a7-820e-2330ed2dbfcd-maas-api-openshift-ca" pod "9669d8a2-2429-43a7-820e-2330ed2dbfcd" (UID: "9669d8a2-2429-43a7-820e-2330ed2dbfcd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:19.779879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:19.779854 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9669d8a2-2429-43a7-820e-2330ed2dbfcd-kube-api-access-7hwg5" pod "9669d8a2-2429-43a7-820e-2330ed2dbfcd" (UID: "9669d8a2-2429-43a7-820e-2330ed2dbfcd"). InnerVolumeSpecName "kube-api-access-7hwg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:19.779879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:19.779866 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9669d8a2-2429-43a7-820e-2330ed2dbfcd-maas-api-xks-ca" pod "9669d8a2-2429-43a7-820e-2330ed2dbfcd" (UID: "9669d8a2-2429-43a7-820e-2330ed2dbfcd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:19.878825 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:19.878770 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9669d8a2-2429-43a7-820e-2330ed2dbfcd-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:19.878825 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:19.878792 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9669d8a2-2429-43a7-820e-2330ed2dbfcd-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:19.878825 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:19.878803 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hwg5\" (UniqueName: \"kubernetes.io/projected/9669d8a2-2429-43a7-820e-2330ed2dbfcd-kube-api-access-7hwg5\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:20.562470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:20.562432 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292" event={"ID":"9669d8a2-2429-43a7-820e-2330ed2dbfcd","Type":"ContainerDied","Data":"4b3b505edf87476037e46db179df051499b0b76e5f6309aae894d302cd2be4b6"} Oct 30 22:40:20.562655 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:20.562481 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b3b505edf87476037e46db179df051499b0b76e5f6309aae894d302cd2be4b6" Oct 30 22:40:20.562655 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:20.562442 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292" Oct 30 22:40:25.320264 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:40:25.320226 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:40:30.680276 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:30.680231 4335 scope.go:122] "RemoveContainer" containerID="ae47b57194803bee7a76210f05fd8404707c2b9bcdd8f84856a5049c7ee82230" Oct 30 22:40:33.850764 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:33.850729 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:40:51.533384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.533273 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf"] Oct 30 22:40:51.534017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.533991 4335 cpu_manager.go:414] "containerMap: removing container" podUID="9669d8a2-2429-43a7-820e-2330ed2dbfcd" containerName="revoke-keys" Oct 30 22:40:51.534114 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.534019 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9669d8a2-2429-43a7-820e-2330ed2dbfcd" containerName="revoke-keys" Oct 30 22:40:51.534202 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.534168 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="9669d8a2-2429-43a7-820e-2330ed2dbfcd" containerName="revoke-keys" Oct 30 22:40:51.539182 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.539152 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf" Oct 30 22:40:51.595146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.595117 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf"] Oct 30 22:40:51.640494 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.640468 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:40:51.640699 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.640526 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:40:51.640699 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.640564 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:40:51.640847 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.640699 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:40:51.685598 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.685573 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rtdt\" (UniqueName: \"kubernetes.io/projected/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-kube-api-access-4rtdt\") pod \"maas-api-revoke-sub-046e1343-4g8kf\" (UID: \"bad96b8f-3162-4656-b2d5-7f0f3d8a138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf" Oct 30 22:40:51.685735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.685631 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-4g8kf\" (UID: \"bad96b8f-3162-4656-b2d5-7f0f3d8a138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf" Oct 30 22:40:51.685823 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.685807 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-4g8kf\" (UID: \"bad96b8f-3162-4656-b2d5-7f0f3d8a138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf" Oct 30 22:40:51.787220 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.787165 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rtdt\" (UniqueName: \"kubernetes.io/projected/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-kube-api-access-4rtdt\") pod \"maas-api-revoke-sub-046e1343-4g8kf\" (UID: \"bad96b8f-3162-4656-b2d5-7f0f3d8a138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf" Oct 30 22:40:51.787220 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.787201 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-4g8kf\" (UID: \"bad96b8f-3162-4656-b2d5-7f0f3d8a138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf" Oct 30 22:40:51.787350 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.787267 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-4g8kf\" (UID: \"bad96b8f-3162-4656-b2d5-7f0f3d8a138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf" Oct 30 22:40:51.788000 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.787981 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-4g8kf\" (UID: \"bad96b8f-3162-4656-b2d5-7f0f3d8a138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf" Oct 30 22:40:51.789823 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.789801 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-4g8kf\" (UID: \"bad96b8f-3162-4656-b2d5-7f0f3d8a138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf" Oct 30 22:40:51.848471 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.848444 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rtdt\" (UniqueName: \"kubernetes.io/projected/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-kube-api-access-4rtdt\") pod \"maas-api-revoke-sub-046e1343-4g8kf\" (UID: \"bad96b8f-3162-4656-b2d5-7f0f3d8a138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf" Oct 30 22:40:51.856711 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:51.856693 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf" Oct 30 22:40:52.232299 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:52.232274 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf"] Oct 30 22:40:52.233716 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:40:52.233681 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad96b8f_3162_4656_b2d5_7f0f3d8a138c.slice/crio-84a083090a89398a65f6ec62ef6411c39837ff3d11c8ed04055fe1c3dfd6730e WatchSource:0}: Error finding container 84a083090a89398a65f6ec62ef6411c39837ff3d11c8ed04055fe1c3dfd6730e: Status 404 returned error can't find the container with id 84a083090a89398a65f6ec62ef6411c39837ff3d11c8ed04055fe1c3dfd6730e Oct 30 22:40:52.779566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:52.779505 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf" event={"ID":"bad96b8f-3162-4656-b2d5-7f0f3d8a138c","Type":"ContainerStarted","Data":"2db9cedb81be70884443853129ea60b546238e4c3e696a81b22e51c3c024a809"} Oct 30 22:40:52.779566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:52.779558 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf" event={"ID":"bad96b8f-3162-4656-b2d5-7f0f3d8a138c","Type":"ContainerStarted","Data":"84a083090a89398a65f6ec62ef6411c39837ff3d11c8ed04055fe1c3dfd6730e"} Oct 30 22:40:52.896989 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:52.896943 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf" podStartSLOduration=1.896928907 podStartE2EDuration="1.896928907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:52.894707132 +0000 UTC m=+3385.995138864" watchObservedRunningTime="2026-10-30 22:40:52.896928907 +0000 UTC m=+3385.997360638" Oct 30 22:40:53.048969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:53.048944 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj"] Oct 30 22:40:53.065500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:53.065476 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3367206a-g9pzj"] Oct 30 22:40:53.790556 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:53.790525 4335 generic.go:356] "Generic (PLEG): container finished" podID="bad96b8f-3162-4656-b2d5-7f0f3d8a138c" containerID="2db9cedb81be70884443853129ea60b546238e4c3e696a81b22e51c3c024a809" exitCode=0 Oct 30 22:40:53.790981 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:53.790639 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf" event={"ID":"bad96b8f-3162-4656-b2d5-7f0f3d8a138c","Type":"ContainerDied","Data":"2db9cedb81be70884443853129ea60b546238e4c3e696a81b22e51c3c024a809"} Oct 30 22:40:54.932198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:54.932173 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf" Oct 30 22:40:55.009592 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:55.009564 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-maas-api-openshift-ca\") pod \"bad96b8f-3162-4656-b2d5-7f0f3d8a138c\" (UID: \"bad96b8f-3162-4656-b2d5-7f0f3d8a138c\") " Oct 30 22:40:55.009717 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:55.009640 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-maas-api-xks-ca\") pod \"bad96b8f-3162-4656-b2d5-7f0f3d8a138c\" (UID: \"bad96b8f-3162-4656-b2d5-7f0f3d8a138c\") " Oct 30 22:40:55.009799 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:55.009751 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-kube-api-access-4rtdt\" (UniqueName: \"kubernetes.io/projected/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-kube-api-access-4rtdt\") pod \"bad96b8f-3162-4656-b2d5-7f0f3d8a138c\" (UID: \"bad96b8f-3162-4656-b2d5-7f0f3d8a138c\") " Oct 30 22:40:55.009867 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:55.009833 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-maas-api-openshift-ca" pod "bad96b8f-3162-4656-b2d5-7f0f3d8a138c" (UID: "bad96b8f-3162-4656-b2d5-7f0f3d8a138c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:55.009977 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:55.009963 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:55.011780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:55.011759 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-maas-api-xks-ca" pod "bad96b8f-3162-4656-b2d5-7f0f3d8a138c" (UID: "bad96b8f-3162-4656-b2d5-7f0f3d8a138c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:55.011901 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:55.011881 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-kube-api-access-4rtdt" pod "bad96b8f-3162-4656-b2d5-7f0f3d8a138c" (UID: "bad96b8f-3162-4656-b2d5-7f0f3d8a138c"). InnerVolumeSpecName "kube-api-access-4rtdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:55.057081 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:55.057055 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc8e93b-8f10-4451-bab6-043704da23ef" path="/var/lib/kubelet/pods/5fc8e93b-8f10-4451-bab6-043704da23ef/volumes" Oct 30 22:40:55.110852 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:55.110829 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rtdt\" (UniqueName: \"kubernetes.io/projected/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-kube-api-access-4rtdt\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:55.110943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:55.110854 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad96b8f-3162-4656-b2d5-7f0f3d8a138c-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:55.320176 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:40:55.320117 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:40:55.804165 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:55.804146 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf" Oct 30 22:40:55.804322 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:55.804140 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf" event={"ID":"bad96b8f-3162-4656-b2d5-7f0f3d8a138c","Type":"ContainerDied","Data":"84a083090a89398a65f6ec62ef6411c39837ff3d11c8ed04055fe1c3dfd6730e"} Oct 30 22:40:55.804322 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:40:55.804266 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84a083090a89398a65f6ec62ef6411c39837ff3d11c8ed04055fe1c3dfd6730e" Oct 30 22:41:01.546807 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.546766 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj"] Oct 30 22:41:01.547369 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.547242 4335 cpu_manager.go:414] "containerMap: removing container" podUID="bad96b8f-3162-4656-b2d5-7f0f3d8a138c" containerName="revoke-keys" Oct 30 22:41:01.547369 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.547256 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad96b8f-3162-4656-b2d5-7f0f3d8a138c" containerName="revoke-keys" Oct 30 22:41:01.547369 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.547349 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="bad96b8f-3162-4656-b2d5-7f0f3d8a138c" containerName="revoke-keys" Oct 30 22:41:01.550817 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.550793 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.558120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.558083 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c7rr\" (UniqueName: \"kubernetes.io/projected/4e6addff-b99f-4e05-a17d-00bb0f88017c-kube-api-access-2c7rr\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.558267 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.558135 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6addff-b99f-4e05-a17d-00bb0f88017c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.558325 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.558278 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6addff-b99f-4e05-a17d-00bb0f88017c-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.558383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.558361 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6addff-b99f-4e05-a17d-00bb0f88017c-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.558431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.558382 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6addff-b99f-4e05-a17d-00bb0f88017c-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.558525 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.558478 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6addff-b99f-4e05-a17d-00bb0f88017c-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.558563 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.558543 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6addff-b99f-4e05-a17d-00bb0f88017c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.558604 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.558592 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6addff-b99f-4e05-a17d-00bb0f88017c-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.558734 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.558713 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6addff-b99f-4e05-a17d-00bb0f88017c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.659607 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.659575 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6addff-b99f-4e05-a17d-00bb0f88017c-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.659764 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.659655 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6addff-b99f-4e05-a17d-00bb0f88017c-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.659764 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.659692 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6addff-b99f-4e05-a17d-00bb0f88017c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.659764 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.659743 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6addff-b99f-4e05-a17d-00bb0f88017c-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.659953 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.659777 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6addff-b99f-4e05-a17d-00bb0f88017c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.659953 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.659864 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c7rr\" (UniqueName: \"kubernetes.io/projected/4e6addff-b99f-4e05-a17d-00bb0f88017c-kube-api-access-2c7rr\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.659953 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.659934 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6addff-b99f-4e05-a17d-00bb0f88017c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.660127 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.659979 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6addff-b99f-4e05-a17d-00bb0f88017c-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.660127 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.660023 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6addff-b99f-4e05-a17d-00bb0f88017c-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.660127 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.660054 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6addff-b99f-4e05-a17d-00bb0f88017c-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.660306 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.660143 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6addff-b99f-4e05-a17d-00bb0f88017c-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.660395 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.660371 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6addff-b99f-4e05-a17d-00bb0f88017c-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.660472 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.660455 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6addff-b99f-4e05-a17d-00bb0f88017c-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.660803 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.660778 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6addff-b99f-4e05-a17d-00bb0f88017c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.662928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.662901 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6addff-b99f-4e05-a17d-00bb0f88017c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.663109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.663089 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6addff-b99f-4e05-a17d-00bb0f88017c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.663935 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.663913 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj"] Oct 30 22:41:01.722544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.722518 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6addff-b99f-4e05-a17d-00bb0f88017c-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.722835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.722813 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c7rr\" (UniqueName: \"kubernetes.io/projected/4e6addff-b99f-4e05-a17d-00bb0f88017c-kube-api-access-2c7rr\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bbbj\" (UID: \"4e6addff-b99f-4e05-a17d-00bb0f88017c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:01.865818 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:01.865793 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:02.295304 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.295281 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj"] Oct 30 22:41:02.296311 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:41:02.296277 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6addff_b99f_4e05_a17d_00bb0f88017c.slice/crio-20809842b45858a1ca4b9176bdc7240840335f34da1b638e57f4c9c6c3a3d7da WatchSource:0}: Error finding container 20809842b45858a1ca4b9176bdc7240840335f34da1b638e57f4c9c6c3a3d7da: Status 404 returned error can't find the container with id 20809842b45858a1ca4b9176bdc7240840335f34da1b638e57f4c9c6c3a3d7da Oct 30 22:41:02.298404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.298370 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:41:02.298519 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.298448 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:41:02.298519 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.298491 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:41:02.386006 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.385978 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8"] Oct 30 22:41:02.389996 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.389964 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8" Oct 30 22:41:02.443355 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.443319 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:41:02.443632 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.443445 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:41:02.443632 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.443571 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:41:02.443775 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.443683 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:41:02.447389 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.447358 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8"] Oct 30 22:41:02.468159 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.468129 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4jmr\" (UniqueName: \"kubernetes.io/projected/ef618a38-7884-47cc-9e1c-74ef6211f88b-kube-api-access-l4jmr\") pod \"maas-api-revoke-sub-d3fba056-57kp8\" (UID: \"ef618a38-7884-47cc-9e1c-74ef6211f88b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8" Oct 30 22:41:02.468312 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.468234 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef618a38-7884-47cc-9e1c-74ef6211f88b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-57kp8\" (UID: \"ef618a38-7884-47cc-9e1c-74ef6211f88b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8" Oct 30 22:41:02.468384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.468308 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef618a38-7884-47cc-9e1c-74ef6211f88b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-57kp8\" (UID: \"ef618a38-7884-47cc-9e1c-74ef6211f88b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8" Oct 30 22:41:02.569461 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.569432 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef618a38-7884-47cc-9e1c-74ef6211f88b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-57kp8\" (UID: \"ef618a38-7884-47cc-9e1c-74ef6211f88b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8" Oct 30 22:41:02.569895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.569510 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef618a38-7884-47cc-9e1c-74ef6211f88b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-57kp8\" (UID: \"ef618a38-7884-47cc-9e1c-74ef6211f88b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8" Oct 30 22:41:02.569895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.569564 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4jmr\" (UniqueName: \"kubernetes.io/projected/ef618a38-7884-47cc-9e1c-74ef6211f88b-kube-api-access-l4jmr\") pod \"maas-api-revoke-sub-d3fba056-57kp8\" (UID: \"ef618a38-7884-47cc-9e1c-74ef6211f88b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8" Oct 30 22:41:02.570197 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.570178 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef618a38-7884-47cc-9e1c-74ef6211f88b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-57kp8\" (UID: \"ef618a38-7884-47cc-9e1c-74ef6211f88b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8" Oct 30 22:41:02.572080 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.572059 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef618a38-7884-47cc-9e1c-74ef6211f88b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-57kp8\" (UID: \"ef618a38-7884-47cc-9e1c-74ef6211f88b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8" Oct 30 22:41:02.630862 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.630832 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4jmr\" (UniqueName: \"kubernetes.io/projected/ef618a38-7884-47cc-9e1c-74ef6211f88b-kube-api-access-l4jmr\") pod \"maas-api-revoke-sub-d3fba056-57kp8\" (UID: \"ef618a38-7884-47cc-9e1c-74ef6211f88b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8" Oct 30 22:41:02.705539 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.705511 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8" Oct 30 22:41:02.852940 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.852857 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" event={"ID":"4e6addff-b99f-4e05-a17d-00bb0f88017c","Type":"ContainerStarted","Data":"b1563e2ba896c7e4d2d27ae31b119bc19d3af14cf8a49a60613589f213dd5d2a"} Oct 30 22:41:02.852940 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.852905 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" event={"ID":"4e6addff-b99f-4e05-a17d-00bb0f88017c","Type":"ContainerStarted","Data":"20809842b45858a1ca4b9176bdc7240840335f34da1b638e57f4c9c6c3a3d7da"} Oct 30 22:41:02.866236 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.866213 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:02.989772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:02.989724 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" podStartSLOduration=1.989710917 podStartE2EDuration="1.989710917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:02.987083935 +0000 UTC m=+3396.087515662" watchObservedRunningTime="2026-10-30 22:41:02.989710917 +0000 UTC m=+3396.090142702" Oct 30 22:41:03.122714 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:03.122601 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8"] Oct 30 22:41:03.127549 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:41:03.127520 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef618a38_7884_47cc_9e1c_74ef6211f88b.slice/crio-6b61e8a1d31324cd14e855121fb4f3f3307e0f1902b9feeb0b79075bb7ba5486 WatchSource:0}: Error finding container 6b61e8a1d31324cd14e855121fb4f3f3307e0f1902b9feeb0b79075bb7ba5486: Status 404 returned error can't find the container with id 6b61e8a1d31324cd14e855121fb4f3f3307e0f1902b9feeb0b79075bb7ba5486 Oct 30 22:41:03.861262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:03.861221 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8" event={"ID":"ef618a38-7884-47cc-9e1c-74ef6211f88b","Type":"ContainerStarted","Data":"367f59fdc44c9823012416498c0bac6d15b64dd69a2391772a22f7d64ccb35c1"} Oct 30 22:41:03.861784 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:03.861329 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8" event={"ID":"ef618a38-7884-47cc-9e1c-74ef6211f88b","Type":"ContainerStarted","Data":"6b61e8a1d31324cd14e855121fb4f3f3307e0f1902b9feeb0b79075bb7ba5486"} Oct 30 22:41:03.868271 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:03.868240 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 22:41:03.868390 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:03.868296 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" podUID="4e6addff-b99f-4e05-a17d-00bb0f88017c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 22:41:03.977880 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:03.977827 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8" podStartSLOduration=1.9778137230000001 podStartE2EDuration="1.977813723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:03.976844102 +0000 UTC m=+3397.077275833" watchObservedRunningTime="2026-10-30 22:41:03.977813723 +0000 UTC m=+3397.078245454" Oct 30 22:41:04.481821 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:04.481728 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:41:04.866944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:04.866912 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 22:41:04.867542 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:04.867507 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" podUID="4e6addff-b99f-4e05-a17d-00bb0f88017c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 22:41:04.872973 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:04.872939 4335 generic.go:356] "Generic (PLEG): container finished" podID="ef618a38-7884-47cc-9e1c-74ef6211f88b" containerID="367f59fdc44c9823012416498c0bac6d15b64dd69a2391772a22f7d64ccb35c1" exitCode=0 Oct 30 22:41:04.873127 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:04.873087 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8" event={"ID":"ef618a38-7884-47cc-9e1c-74ef6211f88b","Type":"ContainerDied","Data":"367f59fdc44c9823012416498c0bac6d15b64dd69a2391772a22f7d64ccb35c1"} Oct 30 22:41:05.866212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:05.866178 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 22:41:05.866504 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:05.866465 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" podUID="4e6addff-b99f-4e05-a17d-00bb0f88017c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 22:41:06.688124 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:06.687801 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8" Oct 30 22:41:06.741652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:06.741161 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef618a38-7884-47cc-9e1c-74ef6211f88b-kube-api-access-l4jmr\" (UniqueName: \"kubernetes.io/projected/ef618a38-7884-47cc-9e1c-74ef6211f88b-kube-api-access-l4jmr\") pod \"ef618a38-7884-47cc-9e1c-74ef6211f88b\" (UID: \"ef618a38-7884-47cc-9e1c-74ef6211f88b\") " Oct 30 22:41:06.741652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:06.741256 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef618a38-7884-47cc-9e1c-74ef6211f88b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef618a38-7884-47cc-9e1c-74ef6211f88b-maas-api-openshift-ca\") pod \"ef618a38-7884-47cc-9e1c-74ef6211f88b\" (UID: \"ef618a38-7884-47cc-9e1c-74ef6211f88b\") " Oct 30 22:41:06.741652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:06.741314 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef618a38-7884-47cc-9e1c-74ef6211f88b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef618a38-7884-47cc-9e1c-74ef6211f88b-maas-api-xks-ca\") pod \"ef618a38-7884-47cc-9e1c-74ef6211f88b\" (UID: \"ef618a38-7884-47cc-9e1c-74ef6211f88b\") " Oct 30 22:41:06.742821 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:06.742775 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef618a38-7884-47cc-9e1c-74ef6211f88b-maas-api-openshift-ca" pod "ef618a38-7884-47cc-9e1c-74ef6211f88b" (UID: "ef618a38-7884-47cc-9e1c-74ef6211f88b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:06.753149 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:06.753101 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef618a38-7884-47cc-9e1c-74ef6211f88b-maas-api-xks-ca" pod "ef618a38-7884-47cc-9e1c-74ef6211f88b" (UID: "ef618a38-7884-47cc-9e1c-74ef6211f88b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:41:06.767811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:06.767755 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef618a38-7884-47cc-9e1c-74ef6211f88b-kube-api-access-l4jmr" pod "ef618a38-7884-47cc-9e1c-74ef6211f88b" (UID: "ef618a38-7884-47cc-9e1c-74ef6211f88b"). InnerVolumeSpecName "kube-api-access-l4jmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:06.849743 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:06.849711 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4jmr\" (UniqueName: \"kubernetes.io/projected/ef618a38-7884-47cc-9e1c-74ef6211f88b-kube-api-access-l4jmr\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:06.850007 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:06.849985 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef618a38-7884-47cc-9e1c-74ef6211f88b-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:06.850150 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:06.850133 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef618a38-7884-47cc-9e1c-74ef6211f88b-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:06.884237 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:06.883012 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 22:41:06.884237 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:06.883084 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" podUID="4e6addff-b99f-4e05-a17d-00bb0f88017c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 22:41:07.193400 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:07.193314 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8" event={"ID":"ef618a38-7884-47cc-9e1c-74ef6211f88b","Type":"ContainerDied","Data":"6b61e8a1d31324cd14e855121fb4f3f3307e0f1902b9feeb0b79075bb7ba5486"} Oct 30 22:41:07.207765 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:07.207741 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b61e8a1d31324cd14e855121fb4f3f3307e0f1902b9feeb0b79075bb7ba5486" Oct 30 22:41:07.207975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:07.193632 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8" Oct 30 22:41:07.866810 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:07.866775 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 22:41:07.867273 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:07.866841 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" podUID="4e6addff-b99f-4e05-a17d-00bb0f88017c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 22:41:08.866292 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:08.866256 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 22:41:08.866477 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:08.866323 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" podUID="4e6addff-b99f-4e05-a17d-00bb0f88017c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 22:41:09.866589 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:09.866555 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 22:41:09.866996 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:09.866626 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" podUID="4e6addff-b99f-4e05-a17d-00bb0f88017c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 22:41:11.908800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:11.908762 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:41:11.909414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:11.909377 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" podUID="4e6addff-b99f-4e05-a17d-00bb0f88017c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:41:12.883194 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:12.883167 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:12.883623 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:12.883592 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:12.884308 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:12.884286 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" Oct 30 22:41:15.239390 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:15.239357 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:41:25.319652 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:41:25.319580 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:41:27.978647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:27.978597 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:41:30.705415 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:30.705380 4335 scope.go:122] "RemoveContainer" containerID="b04c4a388808325c1b38d1506c193274f4cfd5e077292a51660470a539c42968" Oct 30 22:41:37.225480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.225436 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz"] Oct 30 22:41:37.226227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.226204 4335 cpu_manager.go:414] "containerMap: removing container" podUID="ef618a38-7884-47cc-9e1c-74ef6211f88b" containerName="revoke-keys" Oct 30 22:41:37.226275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.226232 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef618a38-7884-47cc-9e1c-74ef6211f88b" containerName="revoke-keys" Oct 30 22:41:37.226431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.226415 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef618a38-7884-47cc-9e1c-74ef6211f88b" containerName="revoke-keys" Oct 30 22:41:37.231588 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.231565 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz" Oct 30 22:41:37.286442 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.286412 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz"] Oct 30 22:41:37.309467 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.309435 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:41:37.331894 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.331867 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:41:37.332010 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.331927 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:41:37.332010 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.331973 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:41:37.395169 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.395142 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94906ed5-5d97-4918-bd5f-77c14a11c8d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-q78xz\" (UID: \"94906ed5-5d97-4918-bd5f-77c14a11c8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz" Oct 30 22:41:37.395274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.395187 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2jkh\" (UniqueName: \"kubernetes.io/projected/94906ed5-5d97-4918-bd5f-77c14a11c8d2-kube-api-access-t2jkh\") pod \"maas-api-revoke-sub-ae594979-q78xz\" (UID: \"94906ed5-5d97-4918-bd5f-77c14a11c8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz" Oct 30 22:41:37.395274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.395261 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94906ed5-5d97-4918-bd5f-77c14a11c8d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-q78xz\" (UID: \"94906ed5-5d97-4918-bd5f-77c14a11c8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz" Oct 30 22:41:37.496527 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.496472 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2jkh\" (UniqueName: \"kubernetes.io/projected/94906ed5-5d97-4918-bd5f-77c14a11c8d2-kube-api-access-t2jkh\") pod \"maas-api-revoke-sub-ae594979-q78xz\" (UID: \"94906ed5-5d97-4918-bd5f-77c14a11c8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz" Oct 30 22:41:37.496527 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.496503 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94906ed5-5d97-4918-bd5f-77c14a11c8d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-q78xz\" (UID: \"94906ed5-5d97-4918-bd5f-77c14a11c8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz" Oct 30 22:41:37.496753 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.496576 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94906ed5-5d97-4918-bd5f-77c14a11c8d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-q78xz\" (UID: \"94906ed5-5d97-4918-bd5f-77c14a11c8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz" Oct 30 22:41:37.497178 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.497151 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94906ed5-5d97-4918-bd5f-77c14a11c8d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-q78xz\" (UID: \"94906ed5-5d97-4918-bd5f-77c14a11c8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz" Oct 30 22:41:37.498989 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.498969 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94906ed5-5d97-4918-bd5f-77c14a11c8d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-q78xz\" (UID: \"94906ed5-5d97-4918-bd5f-77c14a11c8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz" Oct 30 22:41:37.557390 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.557363 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2jkh\" (UniqueName: \"kubernetes.io/projected/94906ed5-5d97-4918-bd5f-77c14a11c8d2-kube-api-access-t2jkh\") pod \"maas-api-revoke-sub-ae594979-q78xz\" (UID: \"94906ed5-5d97-4918-bd5f-77c14a11c8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz" Oct 30 22:41:37.844431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:37.844392 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz" Oct 30 22:41:38.219202 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:38.219173 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz"] Oct 30 22:41:38.220298 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:41:38.220271 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94906ed5_5d97_4918_bd5f_77c14a11c8d2.slice/crio-20bb3ff746462cf1d751d95784fd6f8bb2cb52df4bf7988ce7f3bbdf53f66f60 WatchSource:0}: Error finding container 20bb3ff746462cf1d751d95784fd6f8bb2cb52df4bf7988ce7f3bbdf53f66f60: Status 404 returned error can't find the container with id 20bb3ff746462cf1d751d95784fd6f8bb2cb52df4bf7988ce7f3bbdf53f66f60 Oct 30 22:41:38.414384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:38.414352 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz" event={"ID":"94906ed5-5d97-4918-bd5f-77c14a11c8d2","Type":"ContainerStarted","Data":"20bb3ff746462cf1d751d95784fd6f8bb2cb52df4bf7988ce7f3bbdf53f66f60"} Oct 30 22:41:39.424514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:39.421795 4335 generic.go:356] "Generic (PLEG): container finished" podID="94906ed5-5d97-4918-bd5f-77c14a11c8d2" containerID="8149b3126504586428fe7315a5dfe61e0f8a88a1b27b8ae0f95e3e739e0ed9a5" exitCode=0 Oct 30 22:41:39.424514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:39.422013 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz" event={"ID":"94906ed5-5d97-4918-bd5f-77c14a11c8d2","Type":"ContainerDied","Data":"8149b3126504586428fe7315a5dfe61e0f8a88a1b27b8ae0f95e3e739e0ed9a5"} Oct 30 22:41:40.563103 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:40.563077 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz" Oct 30 22:41:40.724314 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:40.724253 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94906ed5-5d97-4918-bd5f-77c14a11c8d2-kube-api-access-t2jkh\" (UniqueName: \"kubernetes.io/projected/94906ed5-5d97-4918-bd5f-77c14a11c8d2-kube-api-access-t2jkh\") pod \"94906ed5-5d97-4918-bd5f-77c14a11c8d2\" (UID: \"94906ed5-5d97-4918-bd5f-77c14a11c8d2\") " Oct 30 22:41:40.724314 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:40.724286 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94906ed5-5d97-4918-bd5f-77c14a11c8d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94906ed5-5d97-4918-bd5f-77c14a11c8d2-maas-api-xks-ca\") pod \"94906ed5-5d97-4918-bd5f-77c14a11c8d2\" (UID: \"94906ed5-5d97-4918-bd5f-77c14a11c8d2\") " Oct 30 22:41:40.724539 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:40.724324 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94906ed5-5d97-4918-bd5f-77c14a11c8d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94906ed5-5d97-4918-bd5f-77c14a11c8d2-maas-api-openshift-ca\") pod \"94906ed5-5d97-4918-bd5f-77c14a11c8d2\" (UID: \"94906ed5-5d97-4918-bd5f-77c14a11c8d2\") " Oct 30 22:41:40.724784 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:40.724756 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94906ed5-5d97-4918-bd5f-77c14a11c8d2-maas-api-openshift-ca" pod "94906ed5-5d97-4918-bd5f-77c14a11c8d2" (UID: "94906ed5-5d97-4918-bd5f-77c14a11c8d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:40.726506 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:40.726474 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94906ed5-5d97-4918-bd5f-77c14a11c8d2-maas-api-xks-ca" pod "94906ed5-5d97-4918-bd5f-77c14a11c8d2" (UID: "94906ed5-5d97-4918-bd5f-77c14a11c8d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:41:40.726607 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:40.726527 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94906ed5-5d97-4918-bd5f-77c14a11c8d2-kube-api-access-t2jkh" pod "94906ed5-5d97-4918-bd5f-77c14a11c8d2" (UID: "94906ed5-5d97-4918-bd5f-77c14a11c8d2"). InnerVolumeSpecName "kube-api-access-t2jkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:40.825763 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:40.825736 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2jkh\" (UniqueName: \"kubernetes.io/projected/94906ed5-5d97-4918-bd5f-77c14a11c8d2-kube-api-access-t2jkh\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:40.825763 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:40.825760 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94906ed5-5d97-4918-bd5f-77c14a11c8d2-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:40.825899 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:40.825770 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94906ed5-5d97-4918-bd5f-77c14a11c8d2-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:41.435658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:41.435571 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz" Oct 30 22:41:41.435658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:41.435586 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz" event={"ID":"94906ed5-5d97-4918-bd5f-77c14a11c8d2","Type":"ContainerDied","Data":"20bb3ff746462cf1d751d95784fd6f8bb2cb52df4bf7988ce7f3bbdf53f66f60"} Oct 30 22:41:41.435658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:41.435646 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20bb3ff746462cf1d751d95784fd6f8bb2cb52df4bf7988ce7f3bbdf53f66f60" Oct 30 22:41:50.136234 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:41:50.136198 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:41:55.321040 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:41:55.320999 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:42:01.856837 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:01.856798 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:42:08.702347 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.702253 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf"] Oct 30 22:42:08.702976 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.702912 4335 cpu_manager.go:414] "containerMap: removing container" podUID="94906ed5-5d97-4918-bd5f-77c14a11c8d2" containerName="revoke-keys" Oct 30 22:42:08.702976 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.702931 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94906ed5-5d97-4918-bd5f-77c14a11c8d2" containerName="revoke-keys" Oct 30 22:42:08.703104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.703055 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="94906ed5-5d97-4918-bd5f-77c14a11c8d2" containerName="revoke-keys" Oct 30 22:42:08.706985 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.706965 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf" Oct 30 22:42:08.782120 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.782085 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:42:08.794931 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.794901 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1446d169-31d3-4766-bd95-65ff99715212-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kt6sf\" (UID: \"1446d169-31d3-4766-bd95-65ff99715212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf" Oct 30 22:42:08.795077 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.794945 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1446d169-31d3-4766-bd95-65ff99715212-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kt6sf\" (UID: \"1446d169-31d3-4766-bd95-65ff99715212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf" Oct 30 22:42:08.795077 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.795044 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-549jj\" (UniqueName: \"kubernetes.io/projected/1446d169-31d3-4766-bd95-65ff99715212-kube-api-access-549jj\") pod \"maas-api-revoke-sub-a4aef7d7-kt6sf\" (UID: \"1446d169-31d3-4766-bd95-65ff99715212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf" Oct 30 22:42:08.808252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.808231 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:42:08.808417 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.808355 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:42:08.808510 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.808375 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:42:08.811770 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.811751 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf"] Oct 30 22:42:08.896427 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.896401 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-549jj\" (UniqueName: \"kubernetes.io/projected/1446d169-31d3-4766-bd95-65ff99715212-kube-api-access-549jj\") pod \"maas-api-revoke-sub-a4aef7d7-kt6sf\" (UID: \"1446d169-31d3-4766-bd95-65ff99715212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf" Oct 30 22:42:08.896561 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.896502 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1446d169-31d3-4766-bd95-65ff99715212-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kt6sf\" (UID: \"1446d169-31d3-4766-bd95-65ff99715212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf" Oct 30 22:42:08.896561 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.896549 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1446d169-31d3-4766-bd95-65ff99715212-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kt6sf\" (UID: \"1446d169-31d3-4766-bd95-65ff99715212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf" Oct 30 22:42:08.897193 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.897175 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1446d169-31d3-4766-bd95-65ff99715212-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kt6sf\" (UID: \"1446d169-31d3-4766-bd95-65ff99715212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf" Oct 30 22:42:08.898975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.898955 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1446d169-31d3-4766-bd95-65ff99715212-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kt6sf\" (UID: \"1446d169-31d3-4766-bd95-65ff99715212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf" Oct 30 22:42:08.957227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:08.957161 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-549jj\" (UniqueName: \"kubernetes.io/projected/1446d169-31d3-4766-bd95-65ff99715212-kube-api-access-549jj\") pod \"maas-api-revoke-sub-a4aef7d7-kt6sf\" (UID: \"1446d169-31d3-4766-bd95-65ff99715212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf" Oct 30 22:42:09.019993 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:09.019961 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf" Oct 30 22:42:09.398438 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:09.398411 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf"] Oct 30 22:42:09.399408 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:42:09.399378 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1446d169_31d3_4766_bd95_65ff99715212.slice/crio-0888b7109bd766b12f501109809d88ea6110f7a6a07641a3f3db01af7597d053 WatchSource:0}: Error finding container 0888b7109bd766b12f501109809d88ea6110f7a6a07641a3f3db01af7597d053: Status 404 returned error can't find the container with id 0888b7109bd766b12f501109809d88ea6110f7a6a07641a3f3db01af7597d053 Oct 30 22:42:09.652296 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:09.652238 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf" event={"ID":"1446d169-31d3-4766-bd95-65ff99715212","Type":"ContainerStarted","Data":"0888b7109bd766b12f501109809d88ea6110f7a6a07641a3f3db01af7597d053"} Oct 30 22:42:10.659533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:10.659493 4335 generic.go:356] "Generic (PLEG): container finished" podID="1446d169-31d3-4766-bd95-65ff99715212" containerID="2633d100fb1b03e2c5532b88c2076ea63d676c7925cb4932bf2e2c8ec06a1f47" exitCode=0 Oct 30 22:42:10.659886 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:10.659578 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf" event={"ID":"1446d169-31d3-4766-bd95-65ff99715212","Type":"ContainerDied","Data":"2633d100fb1b03e2c5532b88c2076ea63d676c7925cb4932bf2e2c8ec06a1f47"} Oct 30 22:42:11.804495 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:11.804470 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf" Oct 30 22:42:11.818748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:11.818725 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1446d169-31d3-4766-bd95-65ff99715212-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1446d169-31d3-4766-bd95-65ff99715212-maas-api-xks-ca\") pod \"1446d169-31d3-4766-bd95-65ff99715212\" (UID: \"1446d169-31d3-4766-bd95-65ff99715212\") " Oct 30 22:42:11.818843 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:11.818826 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1446d169-31d3-4766-bd95-65ff99715212-kube-api-access-549jj\" (UniqueName: \"kubernetes.io/projected/1446d169-31d3-4766-bd95-65ff99715212-kube-api-access-549jj\") pod \"1446d169-31d3-4766-bd95-65ff99715212\" (UID: \"1446d169-31d3-4766-bd95-65ff99715212\") " Oct 30 22:42:11.818914 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:11.818882 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1446d169-31d3-4766-bd95-65ff99715212-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1446d169-31d3-4766-bd95-65ff99715212-maas-api-openshift-ca\") pod \"1446d169-31d3-4766-bd95-65ff99715212\" (UID: \"1446d169-31d3-4766-bd95-65ff99715212\") " Oct 30 22:42:11.819230 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:11.819208 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1446d169-31d3-4766-bd95-65ff99715212-maas-api-openshift-ca" pod "1446d169-31d3-4766-bd95-65ff99715212" (UID: "1446d169-31d3-4766-bd95-65ff99715212"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:42:11.820881 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:11.820861 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1446d169-31d3-4766-bd95-65ff99715212-maas-api-xks-ca" pod "1446d169-31d3-4766-bd95-65ff99715212" (UID: "1446d169-31d3-4766-bd95-65ff99715212"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:42:11.821052 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:11.821033 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1446d169-31d3-4766-bd95-65ff99715212-kube-api-access-549jj" pod "1446d169-31d3-4766-bd95-65ff99715212" (UID: "1446d169-31d3-4766-bd95-65ff99715212"). InnerVolumeSpecName "kube-api-access-549jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:42:11.919997 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:11.919942 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1446d169-31d3-4766-bd95-65ff99715212-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:42:11.919997 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:11.919968 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-549jj\" (UniqueName: \"kubernetes.io/projected/1446d169-31d3-4766-bd95-65ff99715212-kube-api-access-549jj\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:42:11.919997 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:11.919978 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1446d169-31d3-4766-bd95-65ff99715212-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:42:12.677481 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:12.677448 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf" Oct 30 22:42:12.677687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:12.677476 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf" event={"ID":"1446d169-31d3-4766-bd95-65ff99715212","Type":"ContainerDied","Data":"0888b7109bd766b12f501109809d88ea6110f7a6a07641a3f3db01af7597d053"} Oct 30 22:42:12.677687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:12.677530 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0888b7109bd766b12f501109809d88ea6110f7a6a07641a3f3db01af7597d053" Oct 30 22:42:13.062319 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:13.062291 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj"] Oct 30 22:42:13.065542 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:13.065518 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b97e616a-hrnwj"] Oct 30 22:42:15.056445 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:15.056414 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65dd4857-0bb8-4daf-b312-2202c1b91795" path="/var/lib/kubelet/pods/65dd4857-0bb8-4daf-b312-2202c1b91795/volumes" Oct 30 22:42:21.537271 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:21.537237 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:42:25.322488 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:42:25.322444 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:42:30.731405 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:30.731363 4335 scope.go:122] "RemoveContainer" containerID="26dc134a9e4048ba17efddf5ec5b4293934cd5a514af22c030a69907c44f3da3" Oct 30 22:42:30.742569 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:30.742548 4335 scope.go:122] "RemoveContainer" containerID="2d82618e492818a407b7cc67e939a710edd594ef570e07b681f1bdc88379775d" Oct 30 22:42:30.751862 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:30.751835 4335 scope.go:122] "RemoveContainer" containerID="7bc42aaad84acb70501b1903ba406ebdcd88ca9022c41f3247a598f11b39cf84" Oct 30 22:42:30.761498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:30.761477 4335 scope.go:122] "RemoveContainer" containerID="e4fe8d7212e4f0767c5b10d6b684f52b6cc6fc1d698a40a8f68f67413ed16212" Oct 30 22:42:37.228983 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:37.228951 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:42:40.247640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:40.243708 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7nqtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:42:40.247640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:40.243783 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" podUID="181f13d9-d1ab-43a9-b6a0-7f4a72f7632b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:42:44.800824 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:44.800791 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4"] Oct 30 22:42:44.801296 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:44.801282 4335 cpu_manager.go:414] "containerMap: removing container" podUID="1446d169-31d3-4766-bd95-65ff99715212" containerName="revoke-keys" Oct 30 22:42:44.801343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:44.801298 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1446d169-31d3-4766-bd95-65ff99715212" containerName="revoke-keys" Oct 30 22:42:44.801391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:44.801381 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="1446d169-31d3-4766-bd95-65ff99715212" containerName="revoke-keys" Oct 30 22:42:44.806124 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:44.806103 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4" Oct 30 22:42:44.858736 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:44.858706 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:42:44.858914 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:44.858842 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:42:44.908233 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:44.908198 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:42:44.908377 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:44.908269 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:42:44.911430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:44.911402 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4"] Oct 30 22:42:44.958543 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:44.958510 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgnw6\" (UniqueName: \"kubernetes.io/projected/6c8dd55b-fe53-4aab-8905-0123ca128d49-kube-api-access-fgnw6\") pod \"maas-api-revoke-sub-79805ca9-bclr4\" (UID: \"6c8dd55b-fe53-4aab-8905-0123ca128d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4" Oct 30 22:42:44.958682 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:44.958649 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8dd55b-fe53-4aab-8905-0123ca128d49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-bclr4\" (UID: \"6c8dd55b-fe53-4aab-8905-0123ca128d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4" Oct 30 22:42:44.958752 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:44.958693 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8dd55b-fe53-4aab-8905-0123ca128d49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-bclr4\" (UID: \"6c8dd55b-fe53-4aab-8905-0123ca128d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4" Oct 30 22:42:45.059774 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:45.059715 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgnw6\" (UniqueName: \"kubernetes.io/projected/6c8dd55b-fe53-4aab-8905-0123ca128d49-kube-api-access-fgnw6\") pod \"maas-api-revoke-sub-79805ca9-bclr4\" (UID: \"6c8dd55b-fe53-4aab-8905-0123ca128d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4" Oct 30 22:42:45.059774 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:45.059749 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8dd55b-fe53-4aab-8905-0123ca128d49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-bclr4\" (UID: \"6c8dd55b-fe53-4aab-8905-0123ca128d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4" Oct 30 22:42:45.059774 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:45.059769 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8dd55b-fe53-4aab-8905-0123ca128d49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-bclr4\" (UID: \"6c8dd55b-fe53-4aab-8905-0123ca128d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4" Oct 30 22:42:45.060345 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:45.060325 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8dd55b-fe53-4aab-8905-0123ca128d49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-bclr4\" (UID: \"6c8dd55b-fe53-4aab-8905-0123ca128d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4" Oct 30 22:42:45.062143 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:45.062127 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8dd55b-fe53-4aab-8905-0123ca128d49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-bclr4\" (UID: \"6c8dd55b-fe53-4aab-8905-0123ca128d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4" Oct 30 22:42:45.120926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:45.120899 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgnw6\" (UniqueName: \"kubernetes.io/projected/6c8dd55b-fe53-4aab-8905-0123ca128d49-kube-api-access-fgnw6\") pod \"maas-api-revoke-sub-79805ca9-bclr4\" (UID: \"6c8dd55b-fe53-4aab-8905-0123ca128d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4" Oct 30 22:42:45.420186 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:45.420135 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4" Oct 30 22:42:45.822957 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:45.822929 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4"] Oct 30 22:42:45.823395 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:42:45.823112 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8dd55b_fe53_4aab_8905_0123ca128d49.slice/crio-1f1ba07c735553a7d69dcd11bceebcb3ddaa7dec740e5efc31d744ac11577ac6 WatchSource:0}: Error finding container 1f1ba07c735553a7d69dcd11bceebcb3ddaa7dec740e5efc31d744ac11577ac6: Status 404 returned error can't find the container with id 1f1ba07c735553a7d69dcd11bceebcb3ddaa7dec740e5efc31d744ac11577ac6 Oct 30 22:42:45.923194 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:45.923166 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4" event={"ID":"6c8dd55b-fe53-4aab-8905-0123ca128d49","Type":"ContainerStarted","Data":"1f1ba07c735553a7d69dcd11bceebcb3ddaa7dec740e5efc31d744ac11577ac6"} Oct 30 22:42:46.931071 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:46.931006 4335 generic.go:356] "Generic (PLEG): container finished" podID="6c8dd55b-fe53-4aab-8905-0123ca128d49" containerID="ce728c6009548c3c1a6336275a02d4360858b43fe0c866db719f5689d95d2549" exitCode=0 Oct 30 22:42:46.931425 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:46.931096 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4" event={"ID":"6c8dd55b-fe53-4aab-8905-0123ca128d49","Type":"ContainerDied","Data":"ce728c6009548c3c1a6336275a02d4360858b43fe0c866db719f5689d95d2549"} Oct 30 22:42:48.075090 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:48.075067 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4" Oct 30 22:42:48.189689 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:48.189665 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c8dd55b-fe53-4aab-8905-0123ca128d49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8dd55b-fe53-4aab-8905-0123ca128d49-maas-api-xks-ca\") pod \"6c8dd55b-fe53-4aab-8905-0123ca128d49\" (UID: \"6c8dd55b-fe53-4aab-8905-0123ca128d49\") " Oct 30 22:42:48.189831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:48.189708 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c8dd55b-fe53-4aab-8905-0123ca128d49-kube-api-access-fgnw6\" (UniqueName: \"kubernetes.io/projected/6c8dd55b-fe53-4aab-8905-0123ca128d49-kube-api-access-fgnw6\") pod \"6c8dd55b-fe53-4aab-8905-0123ca128d49\" (UID: \"6c8dd55b-fe53-4aab-8905-0123ca128d49\") " Oct 30 22:42:48.189913 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:48.189835 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c8dd55b-fe53-4aab-8905-0123ca128d49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8dd55b-fe53-4aab-8905-0123ca128d49-maas-api-openshift-ca\") pod \"6c8dd55b-fe53-4aab-8905-0123ca128d49\" (UID: \"6c8dd55b-fe53-4aab-8905-0123ca128d49\") " Oct 30 22:42:48.190589 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:48.190560 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8dd55b-fe53-4aab-8905-0123ca128d49-maas-api-openshift-ca" pod "6c8dd55b-fe53-4aab-8905-0123ca128d49" (UID: "6c8dd55b-fe53-4aab-8905-0123ca128d49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:42:48.192108 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:48.192046 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c8dd55b-fe53-4aab-8905-0123ca128d49-maas-api-xks-ca" pod "6c8dd55b-fe53-4aab-8905-0123ca128d49" (UID: "6c8dd55b-fe53-4aab-8905-0123ca128d49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:42:48.192434 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:48.192388 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8dd55b-fe53-4aab-8905-0123ca128d49-kube-api-access-fgnw6" pod "6c8dd55b-fe53-4aab-8905-0123ca128d49" (UID: "6c8dd55b-fe53-4aab-8905-0123ca128d49"). InnerVolumeSpecName "kube-api-access-fgnw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:42:48.290890 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:48.290868 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8dd55b-fe53-4aab-8905-0123ca128d49-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:42:48.290890 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:48.290890 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgnw6\" (UniqueName: \"kubernetes.io/projected/6c8dd55b-fe53-4aab-8905-0123ca128d49-kube-api-access-fgnw6\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:42:48.291028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:48.290901 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8dd55b-fe53-4aab-8905-0123ca128d49-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:42:48.948508 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:48.948483 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4" Oct 30 22:42:48.948701 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:48.948529 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4" event={"ID":"6c8dd55b-fe53-4aab-8905-0123ca128d49","Type":"ContainerDied","Data":"1f1ba07c735553a7d69dcd11bceebcb3ddaa7dec740e5efc31d744ac11577ac6"} Oct 30 22:42:48.948701 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:48.948602 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f1ba07c735553a7d69dcd11bceebcb3ddaa7dec740e5efc31d744ac11577ac6" Oct 30 22:42:55.320715 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:42:55.320678 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:42:59.438471 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:42:59.438434 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:43:14.048808 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:14.048776 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9"] Oct 30 22:43:14.098252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:14.098221 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ll4h9"] Oct 30 22:43:15.060697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:15.060654 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb65b2a7-047f-4a44-9498-124d804aad75" path="/var/lib/kubelet/pods/cb65b2a7-047f-4a44-9498-124d804aad75/volumes" Oct 30 22:43:25.319457 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:43:25.319419 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:43:28.634686 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.634648 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47"] Oct 30 22:43:28.635098 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.635088 4335 cpu_manager.go:414] "containerMap: removing container" podUID="6c8dd55b-fe53-4aab-8905-0123ca128d49" containerName="revoke-keys" Oct 30 22:43:28.635144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.635101 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c8dd55b-fe53-4aab-8905-0123ca128d49" containerName="revoke-keys" Oct 30 22:43:28.635202 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.635189 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c8dd55b-fe53-4aab-8905-0123ca128d49" containerName="revoke-keys" Oct 30 22:43:28.638646 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.638602 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47" Oct 30 22:43:28.663245 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.663216 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfp7h\" (UniqueName: \"kubernetes.io/projected/93948790-4aa0-439a-871f-778f6089844e-kube-api-access-vfp7h\") pod \"maas-api-revoke-sub-0b7e8edb-btz47\" (UID: \"93948790-4aa0-439a-871f-778f6089844e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47" Oct 30 22:43:28.663381 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.663252 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93948790-4aa0-439a-871f-778f6089844e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-btz47\" (UID: \"93948790-4aa0-439a-871f-778f6089844e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47" Oct 30 22:43:28.663381 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.663281 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93948790-4aa0-439a-871f-778f6089844e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-btz47\" (UID: \"93948790-4aa0-439a-871f-778f6089844e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47" Oct 30 22:43:28.693087 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.693035 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:43:28.742156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.742135 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:43:28.745230 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.745212 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47"] Oct 30 22:43:28.749687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.749668 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:43:28.749687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.749679 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:43:28.763794 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.763772 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfp7h\" (UniqueName: \"kubernetes.io/projected/93948790-4aa0-439a-871f-778f6089844e-kube-api-access-vfp7h\") pod \"maas-api-revoke-sub-0b7e8edb-btz47\" (UID: \"93948790-4aa0-439a-871f-778f6089844e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47" Oct 30 22:43:28.763906 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.763800 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93948790-4aa0-439a-871f-778f6089844e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-btz47\" (UID: \"93948790-4aa0-439a-871f-778f6089844e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47" Oct 30 22:43:28.763906 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.763828 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93948790-4aa0-439a-871f-778f6089844e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-btz47\" (UID: \"93948790-4aa0-439a-871f-778f6089844e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47" Oct 30 22:43:28.764630 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.764597 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93948790-4aa0-439a-871f-778f6089844e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-btz47\" (UID: \"93948790-4aa0-439a-871f-778f6089844e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47" Oct 30 22:43:28.766486 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.766461 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93948790-4aa0-439a-871f-778f6089844e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-btz47\" (UID: \"93948790-4aa0-439a-871f-778f6089844e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47" Oct 30 22:43:28.829348 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.829319 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfp7h\" (UniqueName: \"kubernetes.io/projected/93948790-4aa0-439a-871f-778f6089844e-kube-api-access-vfp7h\") pod \"maas-api-revoke-sub-0b7e8edb-btz47\" (UID: \"93948790-4aa0-439a-871f-778f6089844e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47" Oct 30 22:43:28.952132 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:28.952070 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47" Oct 30 22:43:29.327414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:29.327390 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47"] Oct 30 22:43:29.328459 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:43:29.328433 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93948790_4aa0_439a_871f_778f6089844e.slice/crio-187cea1cbed70cddcb73a4aed11e0b07bc08981b06b7ce96545a7bec3b9d8d85 WatchSource:0}: Error finding container 187cea1cbed70cddcb73a4aed11e0b07bc08981b06b7ce96545a7bec3b9d8d85: Status 404 returned error can't find the container with id 187cea1cbed70cddcb73a4aed11e0b07bc08981b06b7ce96545a7bec3b9d8d85 Oct 30 22:43:30.255544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:30.255518 4335 generic.go:356] "Generic (PLEG): container finished" podID="93948790-4aa0-439a-871f-778f6089844e" containerID="47d7904d21b99130a90b11b25be1c840aa0f4f2d36594e508ca03c6f8e13fc93" exitCode=0 Oct 30 22:43:30.255892 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:30.255594 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47" event={"ID":"93948790-4aa0-439a-871f-778f6089844e","Type":"ContainerDied","Data":"47d7904d21b99130a90b11b25be1c840aa0f4f2d36594e508ca03c6f8e13fc93"} Oct 30 22:43:30.255892 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:30.255653 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47" event={"ID":"93948790-4aa0-439a-871f-778f6089844e","Type":"ContainerStarted","Data":"187cea1cbed70cddcb73a4aed11e0b07bc08981b06b7ce96545a7bec3b9d8d85"} Oct 30 22:43:30.798923 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:30.798893 4335 scope.go:122] "RemoveContainer" containerID="bb599e3e81daf13e736a8cf9f367fee351e92013a019b7d4de1c38c7db013a2a" Oct 30 22:43:30.809532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:30.809509 4335 scope.go:122] "RemoveContainer" containerID="c36b2ad5e446ad4746f72cd8f26ac97205e33ade178001d1d1359213a814126d" Oct 30 22:43:31.424217 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:31.424193 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47" Oct 30 22:43:31.484970 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:31.484946 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93948790-4aa0-439a-871f-778f6089844e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93948790-4aa0-439a-871f-778f6089844e-maas-api-openshift-ca\") pod \"93948790-4aa0-439a-871f-778f6089844e\" (UID: \"93948790-4aa0-439a-871f-778f6089844e\") " Oct 30 22:43:31.485096 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:31.485012 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93948790-4aa0-439a-871f-778f6089844e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93948790-4aa0-439a-871f-778f6089844e-maas-api-xks-ca\") pod \"93948790-4aa0-439a-871f-778f6089844e\" (UID: \"93948790-4aa0-439a-871f-778f6089844e\") " Oct 30 22:43:31.485096 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:31.485040 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93948790-4aa0-439a-871f-778f6089844e-kube-api-access-vfp7h\" (UniqueName: \"kubernetes.io/projected/93948790-4aa0-439a-871f-778f6089844e-kube-api-access-vfp7h\") pod \"93948790-4aa0-439a-871f-778f6089844e\" (UID: \"93948790-4aa0-439a-871f-778f6089844e\") " Oct 30 22:43:31.485295 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:31.485265 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93948790-4aa0-439a-871f-778f6089844e-maas-api-openshift-ca" pod "93948790-4aa0-439a-871f-778f6089844e" (UID: "93948790-4aa0-439a-871f-778f6089844e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:43:31.487162 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:31.487136 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93948790-4aa0-439a-871f-778f6089844e-maas-api-xks-ca" pod "93948790-4aa0-439a-871f-778f6089844e" (UID: "93948790-4aa0-439a-871f-778f6089844e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:43:31.487236 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:31.487165 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93948790-4aa0-439a-871f-778f6089844e-kube-api-access-vfp7h" pod "93948790-4aa0-439a-871f-778f6089844e" (UID: "93948790-4aa0-439a-871f-778f6089844e"). InnerVolumeSpecName "kube-api-access-vfp7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:31.586069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:31.586047 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93948790-4aa0-439a-871f-778f6089844e-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:31.586069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:31.586067 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfp7h\" (UniqueName: \"kubernetes.io/projected/93948790-4aa0-439a-871f-778f6089844e-kube-api-access-vfp7h\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:31.586217 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:31.586079 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93948790-4aa0-439a-871f-778f6089844e-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:32.268708 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:32.268678 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47" event={"ID":"93948790-4aa0-439a-871f-778f6089844e","Type":"ContainerDied","Data":"187cea1cbed70cddcb73a4aed11e0b07bc08981b06b7ce96545a7bec3b9d8d85"} Oct 30 22:43:32.268848 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:32.268724 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="187cea1cbed70cddcb73a4aed11e0b07bc08981b06b7ce96545a7bec3b9d8d85" Oct 30 22:43:32.268848 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:32.268684 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47" Oct 30 22:43:40.007682 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.007596 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw"] Oct 30 22:43:40.008051 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.008021 4335 cpu_manager.go:414] "containerMap: removing container" podUID="93948790-4aa0-439a-871f-778f6089844e" containerName="revoke-keys" Oct 30 22:43:40.008051 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.008032 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93948790-4aa0-439a-871f-778f6089844e" containerName="revoke-keys" Oct 30 22:43:40.008125 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.008106 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="93948790-4aa0-439a-871f-778f6089844e" containerName="revoke-keys" Oct 30 22:43:40.011284 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.011262 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw" Oct 30 22:43:40.059146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.059112 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcld2\" (UniqueName: \"kubernetes.io/projected/490d661c-1615-4554-a23f-0d33cddef639-kube-api-access-pcld2\") pod \"maas-api-revoke-sub-3d4156f1-7pmsw\" (UID: \"490d661c-1615-4554-a23f-0d33cddef639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw" Oct 30 22:43:40.059271 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.059164 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490d661c-1615-4554-a23f-0d33cddef639-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7pmsw\" (UID: \"490d661c-1615-4554-a23f-0d33cddef639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw" Oct 30 22:43:40.059331 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.059263 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490d661c-1615-4554-a23f-0d33cddef639-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7pmsw\" (UID: \"490d661c-1615-4554-a23f-0d33cddef639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw" Oct 30 22:43:40.114314 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.114286 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:43:40.114497 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.114351 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:43:40.114497 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.114365 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:43:40.114497 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.114413 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:43:40.117850 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.117832 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw"] Oct 30 22:43:40.160398 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.160377 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcld2\" (UniqueName: \"kubernetes.io/projected/490d661c-1615-4554-a23f-0d33cddef639-kube-api-access-pcld2\") pod \"maas-api-revoke-sub-3d4156f1-7pmsw\" (UID: \"490d661c-1615-4554-a23f-0d33cddef639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw" Oct 30 22:43:40.160534 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.160409 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490d661c-1615-4554-a23f-0d33cddef639-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7pmsw\" (UID: \"490d661c-1615-4554-a23f-0d33cddef639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw" Oct 30 22:43:40.160534 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.160458 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490d661c-1615-4554-a23f-0d33cddef639-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7pmsw\" (UID: \"490d661c-1615-4554-a23f-0d33cddef639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw" Oct 30 22:43:40.161257 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.161237 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490d661c-1615-4554-a23f-0d33cddef639-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7pmsw\" (UID: \"490d661c-1615-4554-a23f-0d33cddef639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw" Oct 30 22:43:40.162853 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.162835 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490d661c-1615-4554-a23f-0d33cddef639-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7pmsw\" (UID: \"490d661c-1615-4554-a23f-0d33cddef639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw" Oct 30 22:43:40.222066 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.222041 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcld2\" (UniqueName: \"kubernetes.io/projected/490d661c-1615-4554-a23f-0d33cddef639-kube-api-access-pcld2\") pod \"maas-api-revoke-sub-3d4156f1-7pmsw\" (UID: \"490d661c-1615-4554-a23f-0d33cddef639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw" Oct 30 22:43:40.324275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.324256 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw" Oct 30 22:43:40.698779 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:40.698746 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw"] Oct 30 22:43:40.699401 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:43:40.699374 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490d661c_1615_4554_a23f_0d33cddef639.slice/crio-a5f72f53eb57cfc81511ddd90787305e8f07976e813ddd89eac9a9dbc595d0b7 WatchSource:0}: Error finding container a5f72f53eb57cfc81511ddd90787305e8f07976e813ddd89eac9a9dbc595d0b7: Status 404 returned error can't find the container with id a5f72f53eb57cfc81511ddd90787305e8f07976e813ddd89eac9a9dbc595d0b7 Oct 30 22:43:41.325158 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:41.325121 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw" event={"ID":"490d661c-1615-4554-a23f-0d33cddef639","Type":"ContainerStarted","Data":"abbadd6cc9abb8c210ecbf9134984388165c512a598f5a03403159ab3bc4118f"} Oct 30 22:43:41.325643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:41.325173 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw" event={"ID":"490d661c-1615-4554-a23f-0d33cddef639","Type":"ContainerStarted","Data":"a5f72f53eb57cfc81511ddd90787305e8f07976e813ddd89eac9a9dbc595d0b7"} Oct 30 22:43:41.441861 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:41.441820 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw" podStartSLOduration=2.441807226 podStartE2EDuration="2.441807226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:41.440647385 +0000 UTC m=+3554.541079116" watchObservedRunningTime="2026-10-30 22:43:41.441807226 +0000 UTC m=+3554.542238965" Oct 30 22:43:41.512983 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:41.512958 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:43:42.375003 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:42.374964 4335 generic.go:356] "Generic (PLEG): container finished" podID="490d661c-1615-4554-a23f-0d33cddef639" containerID="abbadd6cc9abb8c210ecbf9134984388165c512a598f5a03403159ab3bc4118f" exitCode=0 Oct 30 22:43:42.375687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:42.375657 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw" event={"ID":"490d661c-1615-4554-a23f-0d33cddef639","Type":"ContainerDied","Data":"abbadd6cc9abb8c210ecbf9134984388165c512a598f5a03403159ab3bc4118f"} Oct 30 22:43:43.164646 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:43.163693 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jr7cw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:43:43.164646 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:43.163765 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jr7cw" podUID="bbf808e7-af04-425e-88db-2ebeb7e44240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:43:43.938049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:43.938025 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw" Oct 30 22:43:44.013542 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:44.013469 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/490d661c-1615-4554-a23f-0d33cddef639-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490d661c-1615-4554-a23f-0d33cddef639-maas-api-xks-ca\") pod \"490d661c-1615-4554-a23f-0d33cddef639\" (UID: \"490d661c-1615-4554-a23f-0d33cddef639\") " Oct 30 22:43:44.013542 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:44.013520 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/490d661c-1615-4554-a23f-0d33cddef639-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490d661c-1615-4554-a23f-0d33cddef639-maas-api-openshift-ca\") pod \"490d661c-1615-4554-a23f-0d33cddef639\" (UID: \"490d661c-1615-4554-a23f-0d33cddef639\") " Oct 30 22:43:44.013542 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:44.013540 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/490d661c-1615-4554-a23f-0d33cddef639-kube-api-access-pcld2\" (UniqueName: \"kubernetes.io/projected/490d661c-1615-4554-a23f-0d33cddef639-kube-api-access-pcld2\") pod \"490d661c-1615-4554-a23f-0d33cddef639\" (UID: \"490d661c-1615-4554-a23f-0d33cddef639\") " Oct 30 22:43:44.013876 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:44.013849 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490d661c-1615-4554-a23f-0d33cddef639-maas-api-openshift-ca" pod "490d661c-1615-4554-a23f-0d33cddef639" (UID: "490d661c-1615-4554-a23f-0d33cddef639"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:43:44.015782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:44.015756 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/490d661c-1615-4554-a23f-0d33cddef639-maas-api-xks-ca" pod "490d661c-1615-4554-a23f-0d33cddef639" (UID: "490d661c-1615-4554-a23f-0d33cddef639"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:43:44.015905 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:44.015882 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490d661c-1615-4554-a23f-0d33cddef639-kube-api-access-pcld2" pod "490d661c-1615-4554-a23f-0d33cddef639" (UID: "490d661c-1615-4554-a23f-0d33cddef639"). InnerVolumeSpecName "kube-api-access-pcld2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:44.115038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:44.115013 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490d661c-1615-4554-a23f-0d33cddef639-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:44.115038 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:44.115037 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490d661c-1615-4554-a23f-0d33cddef639-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:44.115214 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:44.115053 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcld2\" (UniqueName: \"kubernetes.io/projected/490d661c-1615-4554-a23f-0d33cddef639-kube-api-access-pcld2\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:44.397563 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:44.397536 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw" Oct 30 22:43:44.397717 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:44.397572 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw" event={"ID":"490d661c-1615-4554-a23f-0d33cddef639","Type":"ContainerDied","Data":"a5f72f53eb57cfc81511ddd90787305e8f07976e813ddd89eac9a9dbc595d0b7"} Oct 30 22:43:44.397717 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:44.397650 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5f72f53eb57cfc81511ddd90787305e8f07976e813ddd89eac9a9dbc595d0b7" Oct 30 22:43:54.560342 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:43:54.560311 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:43:55.337217 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:43:55.337120 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:44:06.935654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:06.935607 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:44:17.498725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.498690 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw"] Oct 30 22:44:17.548443 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.548415 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7pmsw"] Oct 30 22:44:17.622300 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.622261 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92"] Oct 30 22:44:17.622788 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.622769 4335 cpu_manager.go:414] "containerMap: removing container" podUID="490d661c-1615-4554-a23f-0d33cddef639" containerName="revoke-keys" Oct 30 22:44:17.622895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.622788 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="490d661c-1615-4554-a23f-0d33cddef639" containerName="revoke-keys" Oct 30 22:44:17.622956 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.622922 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="490d661c-1615-4554-a23f-0d33cddef639" containerName="revoke-keys" Oct 30 22:44:17.626231 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.626211 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92" Oct 30 22:44:17.680194 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.680173 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:44:17.680351 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.680239 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:44:17.680447 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.680244 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:44:17.680735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.680693 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:44:17.684324 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.684299 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92"] Oct 30 22:44:17.693747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.693726 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8f05ade-8457-4edf-a24d-489973c6c2be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kjx92\" (UID: \"e8f05ade-8457-4edf-a24d-489973c6c2be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92" Oct 30 22:44:17.693857 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.693776 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mghwg\" (UniqueName: \"kubernetes.io/projected/e8f05ade-8457-4edf-a24d-489973c6c2be-kube-api-access-mghwg\") pod \"maas-api-revoke-sub-3d4156f1-kjx92\" (UID: \"e8f05ade-8457-4edf-a24d-489973c6c2be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92" Oct 30 22:44:17.693857 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.693795 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8f05ade-8457-4edf-a24d-489973c6c2be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kjx92\" (UID: \"e8f05ade-8457-4edf-a24d-489973c6c2be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92" Oct 30 22:44:17.794699 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.794609 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8f05ade-8457-4edf-a24d-489973c6c2be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kjx92\" (UID: \"e8f05ade-8457-4edf-a24d-489973c6c2be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92" Oct 30 22:44:17.794835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.794712 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mghwg\" (UniqueName: \"kubernetes.io/projected/e8f05ade-8457-4edf-a24d-489973c6c2be-kube-api-access-mghwg\") pod \"maas-api-revoke-sub-3d4156f1-kjx92\" (UID: \"e8f05ade-8457-4edf-a24d-489973c6c2be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92" Oct 30 22:44:17.794835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.794738 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8f05ade-8457-4edf-a24d-489973c6c2be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kjx92\" (UID: \"e8f05ade-8457-4edf-a24d-489973c6c2be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92" Oct 30 22:44:17.795410 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.795388 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8f05ade-8457-4edf-a24d-489973c6c2be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kjx92\" (UID: \"e8f05ade-8457-4edf-a24d-489973c6c2be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92" Oct 30 22:44:17.797295 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.797271 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8f05ade-8457-4edf-a24d-489973c6c2be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kjx92\" (UID: \"e8f05ade-8457-4edf-a24d-489973c6c2be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92" Oct 30 22:44:17.856172 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.856149 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mghwg\" (UniqueName: \"kubernetes.io/projected/e8f05ade-8457-4edf-a24d-489973c6c2be-kube-api-access-mghwg\") pod \"maas-api-revoke-sub-3d4156f1-kjx92\" (UID: \"e8f05ade-8457-4edf-a24d-489973c6c2be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92" Oct 30 22:44:17.940065 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:17.940043 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92" Oct 30 22:44:18.315414 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:18.315386 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92"] Oct 30 22:44:18.315910 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:44:18.315887 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f05ade_8457_4edf_a24d_489973c6c2be.slice/crio-bc5564560f1630ba0127508914f4ba1a795986b6f5f87c03c08399e023d1fab8 WatchSource:0}: Error finding container bc5564560f1630ba0127508914f4ba1a795986b6f5f87c03c08399e023d1fab8: Status 404 returned error can't find the container with id bc5564560f1630ba0127508914f4ba1a795986b6f5f87c03c08399e023d1fab8 Oct 30 22:44:18.647334 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:18.647310 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92" event={"ID":"e8f05ade-8457-4edf-a24d-489973c6c2be","Type":"ContainerStarted","Data":"88bbaf6e730742e0b5f6f69a96452d61314369ba484f01cc33a74a8c06d19383"} Oct 30 22:44:18.647667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:18.647364 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92" event={"ID":"e8f05ade-8457-4edf-a24d-489973c6c2be","Type":"ContainerStarted","Data":"bc5564560f1630ba0127508914f4ba1a795986b6f5f87c03c08399e023d1fab8"} Oct 30 22:44:18.764199 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:18.764143 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92" podStartSLOduration=1.7641311929999999 podStartE2EDuration="1.764131193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:18.762704799 +0000 UTC m=+3591.863136542" watchObservedRunningTime="2026-10-30 22:44:18.764131193 +0000 UTC m=+3591.864562924" Oct 30 22:44:19.056112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:19.056084 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="490d661c-1615-4554-a23f-0d33cddef639" path="/var/lib/kubelet/pods/490d661c-1615-4554-a23f-0d33cddef639/volumes" Oct 30 22:44:19.654280 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:19.654251 4335 generic.go:356] "Generic (PLEG): container finished" podID="e8f05ade-8457-4edf-a24d-489973c6c2be" containerID="88bbaf6e730742e0b5f6f69a96452d61314369ba484f01cc33a74a8c06d19383" exitCode=0 Oct 30 22:44:19.654719 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:19.654343 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92" event={"ID":"e8f05ade-8457-4edf-a24d-489973c6c2be","Type":"ContainerDied","Data":"88bbaf6e730742e0b5f6f69a96452d61314369ba484f01cc33a74a8c06d19383"} Oct 30 22:44:20.810751 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:20.810724 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92" Oct 30 22:44:20.919919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:20.919867 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8f05ade-8457-4edf-a24d-489973c6c2be-kube-api-access-mghwg\" (UniqueName: \"kubernetes.io/projected/e8f05ade-8457-4edf-a24d-489973c6c2be-kube-api-access-mghwg\") pod \"e8f05ade-8457-4edf-a24d-489973c6c2be\" (UID: \"e8f05ade-8457-4edf-a24d-489973c6c2be\") " Oct 30 22:44:20.919919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:20.919900 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8f05ade-8457-4edf-a24d-489973c6c2be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8f05ade-8457-4edf-a24d-489973c6c2be-maas-api-xks-ca\") pod \"e8f05ade-8457-4edf-a24d-489973c6c2be\" (UID: \"e8f05ade-8457-4edf-a24d-489973c6c2be\") " Oct 30 22:44:20.920104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:20.919923 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8f05ade-8457-4edf-a24d-489973c6c2be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8f05ade-8457-4edf-a24d-489973c6c2be-maas-api-openshift-ca\") pod \"e8f05ade-8457-4edf-a24d-489973c6c2be\" (UID: \"e8f05ade-8457-4edf-a24d-489973c6c2be\") " Oct 30 22:44:20.920328 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:20.920305 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8f05ade-8457-4edf-a24d-489973c6c2be-maas-api-openshift-ca" pod "e8f05ade-8457-4edf-a24d-489973c6c2be" (UID: "e8f05ade-8457-4edf-a24d-489973c6c2be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:44:20.922044 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:20.922010 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8f05ade-8457-4edf-a24d-489973c6c2be-maas-api-xks-ca" pod "e8f05ade-8457-4edf-a24d-489973c6c2be" (UID: "e8f05ade-8457-4edf-a24d-489973c6c2be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:44:20.922146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:20.922105 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f05ade-8457-4edf-a24d-489973c6c2be-kube-api-access-mghwg" pod "e8f05ade-8457-4edf-a24d-489973c6c2be" (UID: "e8f05ade-8457-4edf-a24d-489973c6c2be"). InnerVolumeSpecName "kube-api-access-mghwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:44:21.021113 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:21.021083 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mghwg\" (UniqueName: \"kubernetes.io/projected/e8f05ade-8457-4edf-a24d-489973c6c2be-kube-api-access-mghwg\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:44:21.021113 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:21.021108 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8f05ade-8457-4edf-a24d-489973c6c2be-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:44:21.021304 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:21.021118 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8f05ade-8457-4edf-a24d-489973c6c2be-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:44:21.667574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:21.667517 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92" Oct 30 22:44:21.667574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:21.667561 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92" event={"ID":"e8f05ade-8457-4edf-a24d-489973c6c2be","Type":"ContainerDied","Data":"bc5564560f1630ba0127508914f4ba1a795986b6f5f87c03c08399e023d1fab8"} Oct 30 22:44:21.667760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:21.667607 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc5564560f1630ba0127508914f4ba1a795986b6f5f87c03c08399e023d1fab8" Oct 30 22:44:25.319714 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:44:25.319672 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:44:28.576264 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:28.576237 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:44:28.655184 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:28.655160 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:44:30.839700 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:30.839673 4335 scope.go:122] "RemoveContainer" containerID="cd082cec2f6d9dae3fbbab5f1228531384defdaffcc3b6dc431d036f3b3a7244" Oct 30 22:44:32.235190 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:32.235151 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:44:36.047785 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:36.047750 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd"] Oct 30 22:44:36.097357 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:36.097323 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q8wbd"] Oct 30 22:44:37.056967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:37.056927 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6711338e-7dff-4a43-b994-252eb34f8903" path="/var/lib/kubelet/pods/6711338e-7dff-4a43-b994-252eb34f8903/volumes" Oct 30 22:44:42.510142 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:42.510102 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:44:55.320309 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:44:55.320267 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:44:56.153898 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:56.153861 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:44:58.929649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:58.928743 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bbbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:44:58.929649 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:44:58.928822 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" podUID="4e6addff-b99f-4e05-a17d-00bb0f88017c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:45:00.175835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:00.175804 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7"] Oct 30 22:45:00.176246 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:00.176233 4335 cpu_manager.go:414] "containerMap: removing container" podUID="e8f05ade-8457-4edf-a24d-489973c6c2be" containerName="revoke-keys" Oct 30 22:45:00.176297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:00.176248 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8f05ade-8457-4edf-a24d-489973c6c2be" containerName="revoke-keys" Oct 30 22:45:00.176347 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:00.176338 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8f05ade-8457-4edf-a24d-489973c6c2be" containerName="revoke-keys" Oct 30 22:45:00.180976 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:00.180957 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7" Oct 30 22:45:00.193431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:00.193409 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnnbk\" (UniqueName: \"kubernetes.io/projected/03fbfe45-52ef-474d-83d5-3f96d03400d0-kube-api-access-mnnbk\") pod \"maas-api-key-cleanup-29890005-b7zm7\" (UID: \"03fbfe45-52ef-474d-83d5-3f96d03400d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7" Oct 30 22:45:00.193568 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:00.193489 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03fbfe45-52ef-474d-83d5-3f96d03400d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-b7zm7\" (UID: \"03fbfe45-52ef-474d-83d5-3f96d03400d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7" Oct 30 22:45:00.236669 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:00.236644 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7"] Oct 30 22:45:00.282532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:00.282511 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:45:00.282532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:00.282512 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:45:00.282772 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:00.282581 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:45:00.294584 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:00.294556 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnnbk\" (UniqueName: \"kubernetes.io/projected/03fbfe45-52ef-474d-83d5-3f96d03400d0-kube-api-access-mnnbk\") pod \"maas-api-key-cleanup-29890005-b7zm7\" (UID: \"03fbfe45-52ef-474d-83d5-3f96d03400d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7" Oct 30 22:45:00.294810 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:00.294757 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03fbfe45-52ef-474d-83d5-3f96d03400d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-b7zm7\" (UID: \"03fbfe45-52ef-474d-83d5-3f96d03400d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7" Oct 30 22:45:00.295605 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:00.295555 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03fbfe45-52ef-474d-83d5-3f96d03400d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-b7zm7\" (UID: \"03fbfe45-52ef-474d-83d5-3f96d03400d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7" Oct 30 22:45:00.356017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:00.355995 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnnbk\" (UniqueName: \"kubernetes.io/projected/03fbfe45-52ef-474d-83d5-3f96d03400d0-kube-api-access-mnnbk\") pod \"maas-api-key-cleanup-29890005-b7zm7\" (UID: \"03fbfe45-52ef-474d-83d5-3f96d03400d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7" Oct 30 22:45:00.493904 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:00.493847 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7" Oct 30 22:45:00.870944 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:00.870916 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7"] Oct 30 22:45:00.871952 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:45:00.871928 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03fbfe45_52ef_474d_83d5_3f96d03400d0.slice/crio-3a1724a095951f9d9fe3b6c5144c46d9ad9af979bc7dbe3b6b6b7e2e70e31d65 WatchSource:0}: Error finding container 3a1724a095951f9d9fe3b6c5144c46d9ad9af979bc7dbe3b6b6b7e2e70e31d65: Status 404 returned error can't find the container with id 3a1724a095951f9d9fe3b6c5144c46d9ad9af979bc7dbe3b6b6b7e2e70e31d65 Oct 30 22:45:00.962697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:00.962666 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7" event={"ID":"03fbfe45-52ef-474d-83d5-3f96d03400d0","Type":"ContainerStarted","Data":"3a1724a095951f9d9fe3b6c5144c46d9ad9af979bc7dbe3b6b6b7e2e70e31d65"} Oct 30 22:45:01.970575 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:01.970505 4335 generic.go:356] "Generic (PLEG): container finished" podID="03fbfe45-52ef-474d-83d5-3f96d03400d0" containerID="fd9709a0b0c0e0d50d95008ffbfa14af43f448d8927f72aa53e4c35ba751a440" exitCode=60 Oct 30 22:45:01.970937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:01.970583 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7" event={"ID":"03fbfe45-52ef-474d-83d5-3f96d03400d0","Type":"ContainerDied","Data":"fd9709a0b0c0e0d50d95008ffbfa14af43f448d8927f72aa53e4c35ba751a440"} Oct 30 22:45:01.970937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:01.970790 4335 scope.go:122] "RemoveContainer" containerID="fd9709a0b0c0e0d50d95008ffbfa14af43f448d8927f72aa53e4c35ba751a440" Oct 30 22:45:02.978388 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:02.978356 4335 generic.go:356] "Generic (PLEG): container finished" podID="03fbfe45-52ef-474d-83d5-3f96d03400d0" containerID="c062793df33c82d0e8ea068c46de68be4daa3b38fe32a0e9883a32c31e2b5d5c" exitCode=60 Oct 30 22:45:02.978777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:02.978444 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7" event={"ID":"03fbfe45-52ef-474d-83d5-3f96d03400d0","Type":"ContainerDied","Data":"c062793df33c82d0e8ea068c46de68be4daa3b38fe32a0e9883a32c31e2b5d5c"} Oct 30 22:45:02.978777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:02.978499 4335 scope.go:122] "RemoveContainer" containerID="fd9709a0b0c0e0d50d95008ffbfa14af43f448d8927f72aa53e4c35ba751a440" Oct 30 22:45:02.978777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:02.978647 4335 scope.go:122] "RemoveContainer" containerID="c062793df33c82d0e8ea068c46de68be4daa3b38fe32a0e9883a32c31e2b5d5c" Oct 30 22:45:02.978923 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:45:02.978842 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-b7zm7_odh-ai-gateway-infra(03fbfe45-52ef-474d-83d5-3f96d03400d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7" podUID="03fbfe45-52ef-474d-83d5-3f96d03400d0" Oct 30 22:45:03.986515 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:03.986480 4335 scope.go:122] "RemoveContainer" containerID="c062793df33c82d0e8ea068c46de68be4daa3b38fe32a0e9883a32c31e2b5d5c" Oct 30 22:45:03.986998 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:45:03.986790 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-b7zm7_odh-ai-gateway-infra(03fbfe45-52ef-474d-83d5-3f96d03400d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7" podUID="03fbfe45-52ef-474d-83d5-3f96d03400d0" Oct 30 22:45:08.052401 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:08.052369 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2"] Oct 30 22:45:08.101461 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:08.101435 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-trtl2"] Oct 30 22:45:09.056484 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:09.056445 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a85124c4-eefc-45c3-8a65-bb4b2464446d" path="/var/lib/kubelet/pods/a85124c4-eefc-45c3-8a65-bb4b2464446d/volumes" Oct 30 22:45:17.055431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:17.055398 4335 scope.go:122] "RemoveContainer" containerID="c062793df33c82d0e8ea068c46de68be4daa3b38fe32a0e9883a32c31e2b5d5c" Oct 30 22:45:18.072462 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:18.072433 4335 generic.go:356] "Generic (PLEG): container finished" podID="03fbfe45-52ef-474d-83d5-3f96d03400d0" containerID="06ec10b1103a93884f8e44da696dd7221661c5a5fbcd63b78a4805da80da57af" exitCode=60 Oct 30 22:45:18.072846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:18.072505 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7" event={"ID":"03fbfe45-52ef-474d-83d5-3f96d03400d0","Type":"ContainerDied","Data":"06ec10b1103a93884f8e44da696dd7221661c5a5fbcd63b78a4805da80da57af"} Oct 30 22:45:18.072846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:18.072560 4335 scope.go:122] "RemoveContainer" containerID="c062793df33c82d0e8ea068c46de68be4daa3b38fe32a0e9883a32c31e2b5d5c" Oct 30 22:45:18.072964 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:18.072877 4335 scope.go:122] "RemoveContainer" containerID="06ec10b1103a93884f8e44da696dd7221661c5a5fbcd63b78a4805da80da57af" Oct 30 22:45:18.073088 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:45:18.073068 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-b7zm7_odh-ai-gateway-infra(03fbfe45-52ef-474d-83d5-3f96d03400d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7" podUID="03fbfe45-52ef-474d-83d5-3f96d03400d0" Oct 30 22:45:19.199628 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:19.199582 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7"] Oct 30 22:45:19.334974 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:19.334948 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7" Oct 30 22:45:19.444092 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:19.444064 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03fbfe45-52ef-474d-83d5-3f96d03400d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03fbfe45-52ef-474d-83d5-3f96d03400d0-maas-api-service-ca\") pod \"03fbfe45-52ef-474d-83d5-3f96d03400d0\" (UID: \"03fbfe45-52ef-474d-83d5-3f96d03400d0\") " Oct 30 22:45:19.444245 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:19.444139 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03fbfe45-52ef-474d-83d5-3f96d03400d0-kube-api-access-mnnbk\" (UniqueName: \"kubernetes.io/projected/03fbfe45-52ef-474d-83d5-3f96d03400d0-kube-api-access-mnnbk\") pod \"03fbfe45-52ef-474d-83d5-3f96d03400d0\" (UID: \"03fbfe45-52ef-474d-83d5-3f96d03400d0\") " Oct 30 22:45:19.444470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:19.444445 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03fbfe45-52ef-474d-83d5-3f96d03400d0-maas-api-service-ca" pod "03fbfe45-52ef-474d-83d5-3f96d03400d0" (UID: "03fbfe45-52ef-474d-83d5-3f96d03400d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:19.446239 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:19.446216 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03fbfe45-52ef-474d-83d5-3f96d03400d0-kube-api-access-mnnbk" pod "03fbfe45-52ef-474d-83d5-3f96d03400d0" (UID: "03fbfe45-52ef-474d-83d5-3f96d03400d0"). InnerVolumeSpecName "kube-api-access-mnnbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:19.544868 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:19.544843 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03fbfe45-52ef-474d-83d5-3f96d03400d0-maas-api-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:19.544868 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:19.544867 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnnbk\" (UniqueName: \"kubernetes.io/projected/03fbfe45-52ef-474d-83d5-3f96d03400d0-kube-api-access-mnnbk\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:20.049222 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:20.049186 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292"] Oct 30 22:45:20.052156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:20.052132 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vl292"] Oct 30 22:45:20.087974 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:20.087953 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7" Oct 30 22:45:20.088075 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:20.088062 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7" event={"ID":"03fbfe45-52ef-474d-83d5-3f96d03400d0","Type":"ContainerDied","Data":"3a1724a095951f9d9fe3b6c5144c46d9ad9af979bc7dbe3b6b6b7e2e70e31d65"} Oct 30 22:45:20.088126 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:20.088107 4335 scope.go:122] "RemoveContainer" containerID="06ec10b1103a93884f8e44da696dd7221661c5a5fbcd63b78a4805da80da57af" Oct 30 22:45:20.251373 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:20.251346 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7"] Oct 30 22:45:20.257591 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:20.257568 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-b7zm7"] Oct 30 22:45:21.056156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:21.056126 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03fbfe45-52ef-474d-83d5-3f96d03400d0" path="/var/lib/kubelet/pods/03fbfe45-52ef-474d-83d5-3f96d03400d0/volumes" Oct 30 22:45:21.056536 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:21.056521 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9669d8a2-2429-43a7-820e-2330ed2dbfcd" path="/var/lib/kubelet/pods/9669d8a2-2429-43a7-820e-2330ed2dbfcd/volumes" Oct 30 22:45:24.971268 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:24.971238 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q"] Oct 30 22:45:24.971727 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:24.971710 4335 cpu_manager.go:414] "containerMap: removing container" podUID="03fbfe45-52ef-474d-83d5-3f96d03400d0" containerName="cleanup" Oct 30 22:45:24.971804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:24.971728 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03fbfe45-52ef-474d-83d5-3f96d03400d0" containerName="cleanup" Oct 30 22:45:24.971804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:24.971753 4335 cpu_manager.go:414] "containerMap: removing container" podUID="03fbfe45-52ef-474d-83d5-3f96d03400d0" containerName="cleanup" Oct 30 22:45:24.971804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:24.971758 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03fbfe45-52ef-474d-83d5-3f96d03400d0" containerName="cleanup" Oct 30 22:45:24.971804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:24.971765 4335 cpu_manager.go:414] "containerMap: removing container" podUID="03fbfe45-52ef-474d-83d5-3f96d03400d0" containerName="cleanup" Oct 30 22:45:24.971804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:24.971770 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03fbfe45-52ef-474d-83d5-3f96d03400d0" containerName="cleanup" Oct 30 22:45:24.971982 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:24.971836 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="03fbfe45-52ef-474d-83d5-3f96d03400d0" containerName="cleanup" Oct 30 22:45:24.971982 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:24.971847 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="03fbfe45-52ef-474d-83d5-3f96d03400d0" containerName="cleanup" Oct 30 22:45:24.976306 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:24.976288 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q" Oct 30 22:45:25.028653 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:25.028606 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:45:25.030255 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:25.030228 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:45:25.030391 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:25.030256 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:45:25.078006 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:25.077985 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:45:25.081020 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:25.081003 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q"] Oct 30 22:45:25.089017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:25.089000 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b8k8q\" (UID: \"e8e85d04-bd8b-4d53-95c8-acc90ad279c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q" Oct 30 22:45:25.089346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:25.089326 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qrkn\" (UniqueName: \"kubernetes.io/projected/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-kube-api-access-4qrkn\") pod \"maas-api-revoke-sub-e2fc9c5b-b8k8q\" (UID: \"e8e85d04-bd8b-4d53-95c8-acc90ad279c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q" Oct 30 22:45:25.089394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:25.089376 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b8k8q\" (UID: \"e8e85d04-bd8b-4d53-95c8-acc90ad279c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q" Oct 30 22:45:25.190061 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:25.190040 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b8k8q\" (UID: \"e8e85d04-bd8b-4d53-95c8-acc90ad279c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q" Oct 30 22:45:25.190204 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:25.190088 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qrkn\" (UniqueName: \"kubernetes.io/projected/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-kube-api-access-4qrkn\") pod \"maas-api-revoke-sub-e2fc9c5b-b8k8q\" (UID: \"e8e85d04-bd8b-4d53-95c8-acc90ad279c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q" Oct 30 22:45:25.190204 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:25.190109 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b8k8q\" (UID: \"e8e85d04-bd8b-4d53-95c8-acc90ad279c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q" Oct 30 22:45:25.190728 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:25.190706 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b8k8q\" (UID: \"e8e85d04-bd8b-4d53-95c8-acc90ad279c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q" Oct 30 22:45:25.192570 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:25.192546 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b8k8q\" (UID: \"e8e85d04-bd8b-4d53-95c8-acc90ad279c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q" Oct 30 22:45:25.251002 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:25.250940 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qrkn\" (UniqueName: \"kubernetes.io/projected/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-kube-api-access-4qrkn\") pod \"maas-api-revoke-sub-e2fc9c5b-b8k8q\" (UID: \"e8e85d04-bd8b-4d53-95c8-acc90ad279c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q" Oct 30 22:45:25.288719 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:25.288690 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q" Oct 30 22:45:25.322955 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:45:25.322908 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:45:25.664028 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:25.663992 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q"] Oct 30 22:45:25.664769 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:45:25.664743 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e85d04_bd8b_4d53_95c8_acc90ad279c2.slice/crio-c6aef9e82b9eba406d0c2d28952a946e0380d7b1b7f9d3e4406ef763820a595e WatchSource:0}: Error finding container c6aef9e82b9eba406d0c2d28952a946e0380d7b1b7f9d3e4406ef763820a595e: Status 404 returned error can't find the container with id c6aef9e82b9eba406d0c2d28952a946e0380d7b1b7f9d3e4406ef763820a595e Oct 30 22:45:26.125388 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:26.125357 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q" event={"ID":"e8e85d04-bd8b-4d53-95c8-acc90ad279c2","Type":"ContainerStarted","Data":"04f3f8d8720f16c46b3179c5d02c4a56d386d06ebb07a5c92a95639e1735f034"} Oct 30 22:45:26.125827 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:26.125415 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q" event={"ID":"e8e85d04-bd8b-4d53-95c8-acc90ad279c2","Type":"ContainerStarted","Data":"c6aef9e82b9eba406d0c2d28952a946e0380d7b1b7f9d3e4406ef763820a595e"} Oct 30 22:45:26.248997 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:26.248949 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q" podStartSLOduration=2.248934722 podStartE2EDuration="2.248934722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:26.246162377 +0000 UTC m=+3659.346594108" watchObservedRunningTime="2026-10-30 22:45:26.248934722 +0000 UTC m=+3659.349366460" Oct 30 22:45:27.132848 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:27.132816 4335 generic.go:356] "Generic (PLEG): container finished" podID="e8e85d04-bd8b-4d53-95c8-acc90ad279c2" containerID="04f3f8d8720f16c46b3179c5d02c4a56d386d06ebb07a5c92a95639e1735f034" exitCode=0 Oct 30 22:45:27.133198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:27.132883 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q" event={"ID":"e8e85d04-bd8b-4d53-95c8-acc90ad279c2","Type":"ContainerDied","Data":"04f3f8d8720f16c46b3179c5d02c4a56d386d06ebb07a5c92a95639e1735f034"} Oct 30 22:45:28.273449 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:28.273424 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q" Oct 30 22:45:28.319103 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:28.319077 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-kube-api-access-4qrkn\" (UniqueName: \"kubernetes.io/projected/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-kube-api-access-4qrkn\") pod \"e8e85d04-bd8b-4d53-95c8-acc90ad279c2\" (UID: \"e8e85d04-bd8b-4d53-95c8-acc90ad279c2\") " Oct 30 22:45:28.319243 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:28.319145 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-maas-api-xks-ca\") pod \"e8e85d04-bd8b-4d53-95c8-acc90ad279c2\" (UID: \"e8e85d04-bd8b-4d53-95c8-acc90ad279c2\") " Oct 30 22:45:28.319317 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:28.319240 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-maas-api-openshift-ca\") pod \"e8e85d04-bd8b-4d53-95c8-acc90ad279c2\" (UID: \"e8e85d04-bd8b-4d53-95c8-acc90ad279c2\") " Oct 30 22:45:28.319558 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:28.319537 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-maas-api-openshift-ca" pod "e8e85d04-bd8b-4d53-95c8-acc90ad279c2" (UID: "e8e85d04-bd8b-4d53-95c8-acc90ad279c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:28.321217 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:28.321197 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-kube-api-access-4qrkn" pod "e8e85d04-bd8b-4d53-95c8-acc90ad279c2" (UID: "e8e85d04-bd8b-4d53-95c8-acc90ad279c2"). InnerVolumeSpecName "kube-api-access-4qrkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:28.321300 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:28.321270 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-maas-api-xks-ca" pod "e8e85d04-bd8b-4d53-95c8-acc90ad279c2" (UID: "e8e85d04-bd8b-4d53-95c8-acc90ad279c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:45:28.420427 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:28.420366 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qrkn\" (UniqueName: \"kubernetes.io/projected/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-kube-api-access-4qrkn\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:28.420427 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:28.420392 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:28.420427 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:28.420402 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e85d04-bd8b-4d53-95c8-acc90ad279c2-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:29.147274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:29.147241 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q" event={"ID":"e8e85d04-bd8b-4d53-95c8-acc90ad279c2","Type":"ContainerDied","Data":"c6aef9e82b9eba406d0c2d28952a946e0380d7b1b7f9d3e4406ef763820a595e"} Oct 30 22:45:29.147432 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:29.147286 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6aef9e82b9eba406d0c2d28952a946e0380d7b1b7f9d3e4406ef763820a595e" Oct 30 22:45:29.147432 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:29.147248 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q" Oct 30 22:45:35.742138 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.742098 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q"] Oct 30 22:45:35.742753 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.742733 4335 cpu_manager.go:414] "containerMap: removing container" podUID="e8e85d04-bd8b-4d53-95c8-acc90ad279c2" containerName="revoke-keys" Oct 30 22:45:35.742857 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.742755 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e85d04-bd8b-4d53-95c8-acc90ad279c2" containerName="revoke-keys" Oct 30 22:45:35.742918 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.742886 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="03fbfe45-52ef-474d-83d5-3f96d03400d0" containerName="cleanup" Oct 30 22:45:35.742918 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.742902 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e85d04-bd8b-4d53-95c8-acc90ad279c2" containerName="revoke-keys" Oct 30 22:45:35.746176 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.746146 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q" Oct 30 22:45:35.803563 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.803536 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q"] Oct 30 22:45:35.848792 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.848765 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:45:35.848945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.848840 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:45:35.848945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.848836 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:45:35.848945 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.848890 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:45:35.883430 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.883398 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235088f5-1bbc-4d0c-b6af-5d320ad5448c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-7664q\" (UID: \"235088f5-1bbc-4d0c-b6af-5d320ad5448c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q" Oct 30 22:45:35.883577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.883464 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235088f5-1bbc-4d0c-b6af-5d320ad5448c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-7664q\" (UID: \"235088f5-1bbc-4d0c-b6af-5d320ad5448c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q" Oct 30 22:45:35.883682 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.883597 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v64wj\" (UniqueName: \"kubernetes.io/projected/235088f5-1bbc-4d0c-b6af-5d320ad5448c-kube-api-access-v64wj\") pod \"maas-api-revoke-sub-5a29efb3-7664q\" (UID: \"235088f5-1bbc-4d0c-b6af-5d320ad5448c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q" Oct 30 22:45:35.984741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.984718 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235088f5-1bbc-4d0c-b6af-5d320ad5448c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-7664q\" (UID: \"235088f5-1bbc-4d0c-b6af-5d320ad5448c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q" Oct 30 22:45:35.984889 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.984773 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235088f5-1bbc-4d0c-b6af-5d320ad5448c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-7664q\" (UID: \"235088f5-1bbc-4d0c-b6af-5d320ad5448c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q" Oct 30 22:45:35.984889 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.984800 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v64wj\" (UniqueName: \"kubernetes.io/projected/235088f5-1bbc-4d0c-b6af-5d320ad5448c-kube-api-access-v64wj\") pod \"maas-api-revoke-sub-5a29efb3-7664q\" (UID: \"235088f5-1bbc-4d0c-b6af-5d320ad5448c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q" Oct 30 22:45:35.985333 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.985311 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235088f5-1bbc-4d0c-b6af-5d320ad5448c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-7664q\" (UID: \"235088f5-1bbc-4d0c-b6af-5d320ad5448c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q" Oct 30 22:45:35.987325 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:35.987303 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235088f5-1bbc-4d0c-b6af-5d320ad5448c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-7664q\" (UID: \"235088f5-1bbc-4d0c-b6af-5d320ad5448c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q" Oct 30 22:45:36.044998 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:36.044944 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v64wj\" (UniqueName: \"kubernetes.io/projected/235088f5-1bbc-4d0c-b6af-5d320ad5448c-kube-api-access-v64wj\") pod \"maas-api-revoke-sub-5a29efb3-7664q\" (UID: \"235088f5-1bbc-4d0c-b6af-5d320ad5448c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q" Oct 30 22:45:36.059089 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:36.059071 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q" Oct 30 22:45:36.435572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:36.435542 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q"] Oct 30 22:45:36.437121 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:45:36.437089 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235088f5_1bbc_4d0c_b6af_5d320ad5448c.slice/crio-8a6031712dcb55b86527ad4377ab2f62d47e0502105921c742021be30e24f498 WatchSource:0}: Error finding container 8a6031712dcb55b86527ad4377ab2f62d47e0502105921c742021be30e24f498: Status 404 returned error can't find the container with id 8a6031712dcb55b86527ad4377ab2f62d47e0502105921c742021be30e24f498 Oct 30 22:45:37.195584 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:37.195550 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q" event={"ID":"235088f5-1bbc-4d0c-b6af-5d320ad5448c","Type":"ContainerStarted","Data":"3ae58b923070faf0de44ce0c13ebc526169d056335f37878ab6e6ac19964165b"} Oct 30 22:45:37.195991 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:37.195642 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q" event={"ID":"235088f5-1bbc-4d0c-b6af-5d320ad5448c","Type":"ContainerStarted","Data":"8a6031712dcb55b86527ad4377ab2f62d47e0502105921c742021be30e24f498"} Oct 30 22:45:37.314274 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:37.314219 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q" podStartSLOduration=2.314205191 podStartE2EDuration="2.314205191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:37.311047271 +0000 UTC m=+3670.411479058" watchObservedRunningTime="2026-10-30 22:45:37.314205191 +0000 UTC m=+3670.414636923" Oct 30 22:45:37.943783 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:37.943752 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:45:38.203727 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:38.203649 4335 generic.go:356] "Generic (PLEG): container finished" podID="235088f5-1bbc-4d0c-b6af-5d320ad5448c" containerID="3ae58b923070faf0de44ce0c13ebc526169d056335f37878ab6e6ac19964165b" exitCode=0 Oct 30 22:45:38.203727 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:38.203707 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q" event={"ID":"235088f5-1bbc-4d0c-b6af-5d320ad5448c","Type":"ContainerDied","Data":"3ae58b923070faf0de44ce0c13ebc526169d056335f37878ab6e6ac19964165b"} Oct 30 22:45:40.173211 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:40.172807 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q" Oct 30 22:45:40.237117 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:40.236713 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7nqtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:45:40.237117 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:40.236773 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7nqtp" podUID="181f13d9-d1ab-43a9-b6a0-7f4a72f7632b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:45:40.283638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:40.282930 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/235088f5-1bbc-4d0c-b6af-5d320ad5448c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235088f5-1bbc-4d0c-b6af-5d320ad5448c-maas-api-openshift-ca\") pod \"235088f5-1bbc-4d0c-b6af-5d320ad5448c\" (UID: \"235088f5-1bbc-4d0c-b6af-5d320ad5448c\") " Oct 30 22:45:40.283638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:40.282995 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/235088f5-1bbc-4d0c-b6af-5d320ad5448c-kube-api-access-v64wj\" (UniqueName: \"kubernetes.io/projected/235088f5-1bbc-4d0c-b6af-5d320ad5448c-kube-api-access-v64wj\") pod \"235088f5-1bbc-4d0c-b6af-5d320ad5448c\" (UID: \"235088f5-1bbc-4d0c-b6af-5d320ad5448c\") " Oct 30 22:45:40.283638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:40.283052 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/235088f5-1bbc-4d0c-b6af-5d320ad5448c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235088f5-1bbc-4d0c-b6af-5d320ad5448c-maas-api-xks-ca\") pod \"235088f5-1bbc-4d0c-b6af-5d320ad5448c\" (UID: \"235088f5-1bbc-4d0c-b6af-5d320ad5448c\") " Oct 30 22:45:40.293639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:40.288803 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/235088f5-1bbc-4d0c-b6af-5d320ad5448c-maas-api-openshift-ca" pod "235088f5-1bbc-4d0c-b6af-5d320ad5448c" (UID: "235088f5-1bbc-4d0c-b6af-5d320ad5448c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:40.293639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:40.293290 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235088f5-1bbc-4d0c-b6af-5d320ad5448c-kube-api-access-v64wj" pod "235088f5-1bbc-4d0c-b6af-5d320ad5448c" (UID: "235088f5-1bbc-4d0c-b6af-5d320ad5448c"). InnerVolumeSpecName "kube-api-access-v64wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:40.299655 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:40.298053 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/235088f5-1bbc-4d0c-b6af-5d320ad5448c-maas-api-xks-ca" pod "235088f5-1bbc-4d0c-b6af-5d320ad5448c" (UID: "235088f5-1bbc-4d0c-b6af-5d320ad5448c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:45:40.386987 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:40.386900 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235088f5-1bbc-4d0c-b6af-5d320ad5448c-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:40.387221 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:40.387201 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v64wj\" (UniqueName: \"kubernetes.io/projected/235088f5-1bbc-4d0c-b6af-5d320ad5448c-kube-api-access-v64wj\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:40.387345 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:40.387329 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235088f5-1bbc-4d0c-b6af-5d320ad5448c-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:40.476636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:40.476413 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q" event={"ID":"235088f5-1bbc-4d0c-b6af-5d320ad5448c","Type":"ContainerDied","Data":"8a6031712dcb55b86527ad4377ab2f62d47e0502105921c742021be30e24f498"} Oct 30 22:45:40.476636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:40.476494 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a6031712dcb55b86527ad4377ab2f62d47e0502105921c742021be30e24f498" Oct 30 22:45:40.476636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:40.476580 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q" Oct 30 22:45:41.051791 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:41.051750 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-mjgkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:45:41.052003 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:41.051845 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" podUID="4ed2ae60-d2b4-4ce7-b48b-295a7eabe663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:45:48.650528 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:48.650490 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:45:55.049532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:55.049499 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf"] Oct 30 22:45:55.099069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:55.099037 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4g8kf"] Oct 30 22:45:55.321363 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:45:55.321270 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:45:57.055918 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:45:57.055886 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bad96b8f-3162-4656-b2d5-7f0f3d8a138c" path="/var/lib/kubelet/pods/bad96b8f-3162-4656-b2d5-7f0f3d8a138c/volumes" Oct 30 22:46:02.747501 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:02.747460 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92"] Oct 30 22:46:02.796496 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:02.796466 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjx92"] Oct 30 22:46:02.871769 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:02.871745 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67"] Oct 30 22:46:02.872176 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:02.872163 4335 cpu_manager.go:414] "containerMap: removing container" podUID="235088f5-1bbc-4d0c-b6af-5d320ad5448c" containerName="revoke-keys" Oct 30 22:46:02.872220 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:02.872178 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="235088f5-1bbc-4d0c-b6af-5d320ad5448c" containerName="revoke-keys" Oct 30 22:46:02.872262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:02.872242 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="235088f5-1bbc-4d0c-b6af-5d320ad5448c" containerName="revoke-keys" Oct 30 22:46:02.875390 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:02.875373 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67" Oct 30 22:46:02.929975 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:02.929950 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:46:02.930183 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:02.930149 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:46:02.930469 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:02.930438 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:46:02.930654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:02.930449 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:46:02.933988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:02.933967 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67"] Oct 30 22:46:02.959946 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:02.959925 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkzrz\" (UniqueName: \"kubernetes.io/projected/c4c5709d-563c-48a2-b245-668499bac383-kube-api-access-qkzrz\") pod \"maas-api-revoke-sub-3d4156f1-nvm67\" (UID: \"c4c5709d-563c-48a2-b245-668499bac383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67" Oct 30 22:46:02.960061 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:02.959966 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c5709d-563c-48a2-b245-668499bac383-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nvm67\" (UID: \"c4c5709d-563c-48a2-b245-668499bac383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67" Oct 30 22:46:02.960061 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:02.959991 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c5709d-563c-48a2-b245-668499bac383-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nvm67\" (UID: \"c4c5709d-563c-48a2-b245-668499bac383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67" Oct 30 22:46:03.056714 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:03.056689 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8f05ade-8457-4edf-a24d-489973c6c2be" path="/var/lib/kubelet/pods/e8f05ade-8457-4edf-a24d-489973c6c2be/volumes" Oct 30 22:46:03.060830 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:03.060808 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c5709d-563c-48a2-b245-668499bac383-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nvm67\" (UID: \"c4c5709d-563c-48a2-b245-668499bac383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67" Oct 30 22:46:03.060940 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:03.060920 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzrz\" (UniqueName: \"kubernetes.io/projected/c4c5709d-563c-48a2-b245-668499bac383-kube-api-access-qkzrz\") pod \"maas-api-revoke-sub-3d4156f1-nvm67\" (UID: \"c4c5709d-563c-48a2-b245-668499bac383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67" Oct 30 22:46:03.061015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:03.060981 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c5709d-563c-48a2-b245-668499bac383-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nvm67\" (UID: \"c4c5709d-563c-48a2-b245-668499bac383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67" Oct 30 22:46:03.061609 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:03.061537 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c5709d-563c-48a2-b245-668499bac383-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nvm67\" (UID: \"c4c5709d-563c-48a2-b245-668499bac383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67" Oct 30 22:46:03.063211 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:03.063192 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c5709d-563c-48a2-b245-668499bac383-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nvm67\" (UID: \"c4c5709d-563c-48a2-b245-668499bac383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67" Oct 30 22:46:03.122280 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:03.122257 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkzrz\" (UniqueName: \"kubernetes.io/projected/c4c5709d-563c-48a2-b245-668499bac383-kube-api-access-qkzrz\") pod \"maas-api-revoke-sub-3d4156f1-nvm67\" (UID: \"c4c5709d-563c-48a2-b245-668499bac383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67" Oct 30 22:46:03.188122 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:03.188105 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67" Oct 30 22:46:03.568771 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:03.568744 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67"] Oct 30 22:46:03.569244 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:46:03.569220 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c5709d_563c_48a2_b245_668499bac383.slice/crio-64540e5028f05f3bdb8874c314586499690f0194e1a98134cf80ed90d7de5e10 WatchSource:0}: Error finding container 64540e5028f05f3bdb8874c314586499690f0194e1a98134cf80ed90d7de5e10: Status 404 returned error can't find the container with id 64540e5028f05f3bdb8874c314586499690f0194e1a98134cf80ed90d7de5e10 Oct 30 22:46:03.668953 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:03.668911 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67" event={"ID":"c4c5709d-563c-48a2-b245-668499bac383","Type":"ContainerStarted","Data":"64540e5028f05f3bdb8874c314586499690f0194e1a98134cf80ed90d7de5e10"} Oct 30 22:46:04.676920 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:04.676849 4335 generic.go:356] "Generic (PLEG): container finished" podID="c4c5709d-563c-48a2-b245-668499bac383" containerID="3ea3ceef4284bc444cca11b0394e750ae01640274367dd25dcb5c48bd73fb772" exitCode=0 Oct 30 22:46:04.677249 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:04.676938 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67" event={"ID":"c4c5709d-563c-48a2-b245-668499bac383","Type":"ContainerDied","Data":"3ea3ceef4284bc444cca11b0394e750ae01640274367dd25dcb5c48bd73fb772"} Oct 30 22:46:05.816054 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:05.816029 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67" Oct 30 22:46:05.983220 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:05.983155 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4c5709d-563c-48a2-b245-668499bac383-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c5709d-563c-48a2-b245-668499bac383-maas-api-openshift-ca\") pod \"c4c5709d-563c-48a2-b245-668499bac383\" (UID: \"c4c5709d-563c-48a2-b245-668499bac383\") " Oct 30 22:46:05.983220 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:05.983204 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4c5709d-563c-48a2-b245-668499bac383-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c5709d-563c-48a2-b245-668499bac383-maas-api-xks-ca\") pod \"c4c5709d-563c-48a2-b245-668499bac383\" (UID: \"c4c5709d-563c-48a2-b245-668499bac383\") " Oct 30 22:46:05.983399 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:05.983240 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4c5709d-563c-48a2-b245-668499bac383-kube-api-access-qkzrz\" (UniqueName: \"kubernetes.io/projected/c4c5709d-563c-48a2-b245-668499bac383-kube-api-access-qkzrz\") pod \"c4c5709d-563c-48a2-b245-668499bac383\" (UID: \"c4c5709d-563c-48a2-b245-668499bac383\") " Oct 30 22:46:05.983518 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:05.983498 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4c5709d-563c-48a2-b245-668499bac383-maas-api-openshift-ca" pod "c4c5709d-563c-48a2-b245-668499bac383" (UID: "c4c5709d-563c-48a2-b245-668499bac383"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:46:05.985464 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:05.985435 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4c5709d-563c-48a2-b245-668499bac383-maas-api-xks-ca" pod "c4c5709d-563c-48a2-b245-668499bac383" (UID: "c4c5709d-563c-48a2-b245-668499bac383"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:46:05.985607 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:05.985476 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4c5709d-563c-48a2-b245-668499bac383-kube-api-access-qkzrz" pod "c4c5709d-563c-48a2-b245-668499bac383" (UID: "c4c5709d-563c-48a2-b245-668499bac383"). InnerVolumeSpecName "kube-api-access-qkzrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:46:06.048395 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:06.048369 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8"] Oct 30 22:46:06.065867 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:06.065844 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-57kp8"] Oct 30 22:46:06.084397 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:06.084375 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c5709d-563c-48a2-b245-668499bac383-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:46:06.084492 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:06.084400 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c5709d-563c-48a2-b245-668499bac383-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:46:06.084492 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:06.084415 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkzrz\" (UniqueName: \"kubernetes.io/projected/c4c5709d-563c-48a2-b245-668499bac383-kube-api-access-qkzrz\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:46:06.690281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:06.690250 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67" event={"ID":"c4c5709d-563c-48a2-b245-668499bac383","Type":"ContainerDied","Data":"64540e5028f05f3bdb8874c314586499690f0194e1a98134cf80ed90d7de5e10"} Oct 30 22:46:06.690444 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:06.690322 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64540e5028f05f3bdb8874c314586499690f0194e1a98134cf80ed90d7de5e10" Oct 30 22:46:06.690444 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:06.690262 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67" Oct 30 22:46:07.055993 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:07.055959 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef618a38-7884-47cc-9e1c-74ef6211f88b" path="/var/lib/kubelet/pods/ef618a38-7884-47cc-9e1c-74ef6211f88b/volumes" Oct 30 22:46:17.551811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:17.551781 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:46:25.319984 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:46:25.319940 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:46:28.372294 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:28.372259 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:46:30.882813 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:30.882782 4335 scope.go:122] "RemoveContainer" containerID="4b02e55fbce358cdcd6b778c18d8459a0a5ecc72a1c11cdec0f286ca28ca99b4" Oct 30 22:46:30.895950 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:30.895918 4335 scope.go:122] "RemoveContainer" containerID="ea680fa5b775d77a837bccfef06c8fc9a81603e085008ee4e4789dbd2d324e70" Oct 30 22:46:30.904555 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:30.904535 4335 scope.go:122] "RemoveContainer" containerID="c84470a93c55c8e2fae45dabe3a001f0109005f94499480a47d1f2eebde7c660" Oct 30 22:46:41.071309 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:41.071221 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz"] Oct 30 22:46:41.119332 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:41.119298 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-q78xz"] Oct 30 22:46:43.059532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:43.059492 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94906ed5-5d97-4918-bd5f-77c14a11c8d2" path="/var/lib/kubelet/pods/94906ed5-5d97-4918-bd5f-77c14a11c8d2/volumes" Oct 30 22:46:43.409282 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:43.409205 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk"] Oct 30 22:46:43.409667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:43.409652 4335 cpu_manager.go:414] "containerMap: removing container" podUID="c4c5709d-563c-48a2-b245-668499bac383" containerName="revoke-keys" Oct 30 22:46:43.409730 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:43.409668 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4c5709d-563c-48a2-b245-668499bac383" containerName="revoke-keys" Oct 30 22:46:43.409782 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:43.409748 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4c5709d-563c-48a2-b245-668499bac383" containerName="revoke-keys" Oct 30 22:46:43.413080 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:43.413060 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" Oct 30 22:46:43.450514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:43.450490 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95f3db1-460d-4169-845e-af21a6e21e1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk\" (UID: \"c95f3db1-460d-4169-845e-af21a6e21e1e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" Oct 30 22:46:43.450642 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:43.450519 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkshm\" (UniqueName: \"kubernetes.io/projected/c95f3db1-460d-4169-845e-af21a6e21e1e-kube-api-access-bkshm\") pod \"payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk\" (UID: \"c95f3db1-460d-4169-845e-af21a6e21e1e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" Oct 30 22:46:43.472501 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:43.472473 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk"] Oct 30 22:46:43.515939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:43.515899 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-5ef07a66-dockercfg-vrksx\"" Oct 30 22:46:43.515939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:43.515900 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-d0a613\"" Oct 30 22:46:43.551760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:43.551734 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95f3db1-460d-4169-845e-af21a6e21e1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk\" (UID: \"c95f3db1-460d-4169-845e-af21a6e21e1e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" Oct 30 22:46:43.551903 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:43.551766 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkshm\" (UniqueName: \"kubernetes.io/projected/c95f3db1-460d-4169-845e-af21a6e21e1e-kube-api-access-bkshm\") pod \"payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk\" (UID: \"c95f3db1-460d-4169-845e-af21a6e21e1e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" Oct 30 22:46:43.552447 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:43.552428 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95f3db1-460d-4169-845e-af21a6e21e1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk\" (UID: \"c95f3db1-460d-4169-845e-af21a6e21e1e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" Oct 30 22:46:43.612994 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:43.612968 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkshm\" (UniqueName: \"kubernetes.io/projected/c95f3db1-460d-4169-845e-af21a6e21e1e-kube-api-access-bkshm\") pod \"payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk\" (UID: \"c95f3db1-460d-4169-845e-af21a6e21e1e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" Oct 30 22:46:43.726709 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:43.726648 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" Oct 30 22:46:44.104011 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:44.103984 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk"] Oct 30 22:46:44.104411 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:46:44.104371 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95f3db1_460d_4169_845e_af21a6e21e1e.slice/crio-064849c35de34bfe42cdcea0c92ce643281cccfd7e92c5465be52f33e516603e WatchSource:0}: Error finding container 064849c35de34bfe42cdcea0c92ce643281cccfd7e92c5465be52f33e516603e: Status 404 returned error can't find the container with id 064849c35de34bfe42cdcea0c92ce643281cccfd7e92c5465be52f33e516603e Oct 30 22:46:44.951698 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:44.951668 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" event={"ID":"c95f3db1-460d-4169-845e-af21a6e21e1e","Type":"ContainerStarted","Data":"9a4a3b6e45ff47d2456f97d055c50dde7cb697c36e8b15ed731c5f0566507da1"} Oct 30 22:46:44.951896 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:44.951715 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" event={"ID":"c95f3db1-460d-4169-845e-af21a6e21e1e","Type":"ContainerStarted","Data":"064849c35de34bfe42cdcea0c92ce643281cccfd7e92c5465be52f33e516603e"} Oct 30 22:46:44.951896 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:44.951767 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" Oct 30 22:46:45.069166 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:45.069118 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" podStartSLOduration=2.069107157 podStartE2EDuration="2.069107157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:45.06678132 +0000 UTC m=+3738.167213053" watchObservedRunningTime="2026-10-30 22:46:45.069107157 +0000 UTC m=+3738.169538889" Oct 30 22:46:54.298403 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:54.298368 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:46:55.322301 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:46:55.322262 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:46:55.967049 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:46:55.967020 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" Oct 30 22:47:12.051128 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:12.051092 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf"] Oct 30 22:47:12.100712 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:12.100672 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt6sf"] Oct 30 22:47:13.056286 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:13.056246 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1446d169-31d3-4766-bd95-65ff99715212" path="/var/lib/kubelet/pods/1446d169-31d3-4766-bd95-65ff99715212/volumes" Oct 30 22:47:23.820764 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:23.820734 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh"] Oct 30 22:47:23.824394 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:23.824375 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:23.929212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:23.929185 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d0a613\"/\"kube-root-ca.crt\"" Oct 30 22:47:23.929212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:23.929202 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d0a613\"/\"default-dockercfg-j5zrt\"" Oct 30 22:47:23.929212 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:23.929185 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d0a613\"/\"simulator-single-main-d0a613-kserve-self-signed-certs\"" Oct 30 22:47:23.929460 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:23.929195 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d0a613\"/\"openshift-service-ca.crt\"" Oct 30 22:47:23.932214 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:23.932195 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh"] Oct 30 22:47:23.996113 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:23.996083 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-tmp-dir\") pod \"simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:23.996318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:23.996121 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slvkl\" (UniqueName: \"kubernetes.io/projected/1f7e91d1-07f1-47b9-ba9a-a9e219957752-kube-api-access-slvkl\") pod \"simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:23.996318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:23.996225 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-home\") pod \"simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:23.996318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:23.996250 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-dshm\") pod \"simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:23.996318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:23.996267 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f7e91d1-07f1-47b9-ba9a-a9e219957752-tls-certs\") pod \"simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:23.996318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:23.996286 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-model-cache\") pod \"simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:24.097229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:24.097162 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-home\") pod \"simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:24.097229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:24.097195 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-dshm\") pod \"simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:24.097229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:24.097210 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f7e91d1-07f1-47b9-ba9a-a9e219957752-tls-certs\") pod \"simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:24.097229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:24.097226 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-model-cache\") pod \"simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:24.097536 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:24.097294 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-tmp-dir\") pod \"simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:24.097536 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:24.097311 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slvkl\" (UniqueName: \"kubernetes.io/projected/1f7e91d1-07f1-47b9-ba9a-a9e219957752-kube-api-access-slvkl\") pod \"simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:24.097780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:24.097755 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-model-cache\") pod \"simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:24.097847 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:24.097776 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-home\") pod \"simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:24.097847 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:24.097833 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-tmp-dir\") pod \"simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:24.099589 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:24.099568 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-dshm\") pod \"simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:24.099954 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:24.099936 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f7e91d1-07f1-47b9-ba9a-a9e219957752-tls-certs\") pod \"simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:24.158057 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:24.158031 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slvkl\" (UniqueName: \"kubernetes.io/projected/1f7e91d1-07f1-47b9-ba9a-a9e219957752-kube-api-access-slvkl\") pod \"simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:24.436941 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:24.436862 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:24.818747 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:24.818720 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh"] Oct 30 22:47:24.820243 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:47:24.820210 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f7e91d1_07f1_47b9_ba9a_a9e219957752.slice/crio-e27b5591716440b7d53f3f4b31098709e88f89393e4c4a12dd3bd13bd8155a49 WatchSource:0}: Error finding container e27b5591716440b7d53f3f4b31098709e88f89393e4c4a12dd3bd13bd8155a49: Status 404 returned error can't find the container with id e27b5591716440b7d53f3f4b31098709e88f89393e4c4a12dd3bd13bd8155a49 Oct 30 22:47:25.212590 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:25.212506 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" event={"ID":"1f7e91d1-07f1-47b9-ba9a-a9e219957752","Type":"ContainerStarted","Data":"deffd2b6abd2299b73772c7b23bf85890f1ee933684673ba6fbc9b993a6f9f19"} Oct 30 22:47:25.213086 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:25.213060 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" event={"ID":"1f7e91d1-07f1-47b9-ba9a-a9e219957752","Type":"ContainerStarted","Data":"e27b5591716440b7d53f3f4b31098709e88f89393e4c4a12dd3bd13bd8155a49"} Oct 30 22:47:25.320732 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:47:25.320697 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:47:25.330881 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:25.330839 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" podStartSLOduration=2.3308256050000002 podStartE2EDuration="2.330825605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:25.32973121 +0000 UTC m=+3778.430162940" watchObservedRunningTime="2026-10-30 22:47:25.330825605 +0000 UTC m=+3778.431257336" Oct 30 22:47:25.437257 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:25.437231 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:28.616500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:28.616466 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:47:30.934580 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:30.934540 4335 scope.go:122] "RemoveContainer" containerID="367f59fdc44c9823012416498c0bac6d15b64dd69a2391772a22f7d64ccb35c1" Oct 30 22:47:30.944155 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:30.944134 4335 scope.go:122] "RemoveContainer" containerID="2db9cedb81be70884443853129ea60b546238e4c3e696a81b22e51c3c024a809" Oct 30 22:47:34.437859 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:34.437828 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:34.450531 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:34.450503 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:35.284563 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:35.284528 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:47:49.049103 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:49.049067 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4"] Oct 30 22:47:49.097921 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:49.097894 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bclr4"] Oct 30 22:47:51.056721 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:51.056689 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c8dd55b-fe53-4aab-8905-0123ca128d49" path="/var/lib/kubelet/pods/6c8dd55b-fe53-4aab-8905-0123ca128d49/volumes" Oct 30 22:47:53.447308 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:47:53.447277 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:47:55.320145 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:47:55.320107 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:48:07.849140 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:07.849107 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765"] Oct 30 22:48:07.852406 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:07.852372 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765" Oct 30 22:48:07.907404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:07.907371 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:48:07.956842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:07.956820 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:48:07.956981 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:07.956868 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:48:07.956981 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:07.956917 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:48:07.961483 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:07.961460 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765"] Oct 30 22:48:07.981255 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:07.981232 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c502b64e-586e-4b56-9da1-73a4173c3084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fac3c348-p7765\" (UID: \"c502b64e-586e-4b56-9da1-73a4173c3084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765" Oct 30 22:48:07.981359 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:07.981304 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tcjg\" (UniqueName: \"kubernetes.io/projected/c502b64e-586e-4b56-9da1-73a4173c3084-kube-api-access-4tcjg\") pod \"maas-api-revoke-sub-fac3c348-p7765\" (UID: \"c502b64e-586e-4b56-9da1-73a4173c3084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765" Oct 30 22:48:07.981416 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:07.981369 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c502b64e-586e-4b56-9da1-73a4173c3084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fac3c348-p7765\" (UID: \"c502b64e-586e-4b56-9da1-73a4173c3084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765" Oct 30 22:48:08.082410 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:08.082389 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tcjg\" (UniqueName: \"kubernetes.io/projected/c502b64e-586e-4b56-9da1-73a4173c3084-kube-api-access-4tcjg\") pod \"maas-api-revoke-sub-fac3c348-p7765\" (UID: \"c502b64e-586e-4b56-9da1-73a4173c3084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765" Oct 30 22:48:08.082523 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:08.082421 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c502b64e-586e-4b56-9da1-73a4173c3084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fac3c348-p7765\" (UID: \"c502b64e-586e-4b56-9da1-73a4173c3084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765" Oct 30 22:48:08.082523 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:08.082457 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c502b64e-586e-4b56-9da1-73a4173c3084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fac3c348-p7765\" (UID: \"c502b64e-586e-4b56-9da1-73a4173c3084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765" Oct 30 22:48:08.083091 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:08.083073 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c502b64e-586e-4b56-9da1-73a4173c3084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fac3c348-p7765\" (UID: \"c502b64e-586e-4b56-9da1-73a4173c3084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765" Oct 30 22:48:08.084903 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:08.084880 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c502b64e-586e-4b56-9da1-73a4173c3084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fac3c348-p7765\" (UID: \"c502b64e-586e-4b56-9da1-73a4173c3084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765" Oct 30 22:48:08.143682 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:08.143629 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tcjg\" (UniqueName: \"kubernetes.io/projected/c502b64e-586e-4b56-9da1-73a4173c3084-kube-api-access-4tcjg\") pod \"maas-api-revoke-sub-fac3c348-p7765\" (UID: \"c502b64e-586e-4b56-9da1-73a4173c3084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765" Oct 30 22:48:08.166249 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:08.166223 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765" Oct 30 22:48:08.541094 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:08.541063 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765"] Oct 30 22:48:08.541829 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:48:08.541801 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc502b64e_586e_4b56_9da1_73a4173c3084.slice/crio-b00bd95b94d2894995ea0c007f29565e9da34adf0551649911b72074ea9844c8 WatchSource:0}: Error finding container b00bd95b94d2894995ea0c007f29565e9da34adf0551649911b72074ea9844c8: Status 404 returned error can't find the container with id b00bd95b94d2894995ea0c007f29565e9da34adf0551649911b72074ea9844c8 Oct 30 22:48:09.487500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:09.487456 4335 generic.go:356] "Generic (PLEG): container finished" podID="c502b64e-586e-4b56-9da1-73a4173c3084" containerID="5546746cbef39850fdac379616c73f3a68498a069563a8fc3069715a2ca2299d" exitCode=0 Oct 30 22:48:09.488109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:09.487521 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765" event={"ID":"c502b64e-586e-4b56-9da1-73a4173c3084","Type":"ContainerDied","Data":"5546746cbef39850fdac379616c73f3a68498a069563a8fc3069715a2ca2299d"} Oct 30 22:48:09.488109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:09.487584 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765" event={"ID":"c502b64e-586e-4b56-9da1-73a4173c3084","Type":"ContainerStarted","Data":"b00bd95b94d2894995ea0c007f29565e9da34adf0551649911b72074ea9844c8"} Oct 30 22:48:10.635905 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:10.635879 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765" Oct 30 22:48:10.706994 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:10.706967 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c502b64e-586e-4b56-9da1-73a4173c3084-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c502b64e-586e-4b56-9da1-73a4173c3084-maas-api-xks-ca\") pod \"c502b64e-586e-4b56-9da1-73a4173c3084\" (UID: \"c502b64e-586e-4b56-9da1-73a4173c3084\") " Oct 30 22:48:10.707098 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:10.707029 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c502b64e-586e-4b56-9da1-73a4173c3084-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c502b64e-586e-4b56-9da1-73a4173c3084-maas-api-openshift-ca\") pod \"c502b64e-586e-4b56-9da1-73a4173c3084\" (UID: \"c502b64e-586e-4b56-9da1-73a4173c3084\") " Oct 30 22:48:10.707098 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:10.707054 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c502b64e-586e-4b56-9da1-73a4173c3084-kube-api-access-4tcjg\" (UniqueName: \"kubernetes.io/projected/c502b64e-586e-4b56-9da1-73a4173c3084-kube-api-access-4tcjg\") pod \"c502b64e-586e-4b56-9da1-73a4173c3084\" (UID: \"c502b64e-586e-4b56-9da1-73a4173c3084\") " Oct 30 22:48:10.707365 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:10.707344 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c502b64e-586e-4b56-9da1-73a4173c3084-maas-api-openshift-ca" pod "c502b64e-586e-4b56-9da1-73a4173c3084" (UID: "c502b64e-586e-4b56-9da1-73a4173c3084"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:48:10.707431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:10.707411 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c502b64e-586e-4b56-9da1-73a4173c3084-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:10.709157 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:10.709135 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c502b64e-586e-4b56-9da1-73a4173c3084-maas-api-xks-ca" pod "c502b64e-586e-4b56-9da1-73a4173c3084" (UID: "c502b64e-586e-4b56-9da1-73a4173c3084"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:48:10.709288 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:10.709270 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c502b64e-586e-4b56-9da1-73a4173c3084-kube-api-access-4tcjg" pod "c502b64e-586e-4b56-9da1-73a4173c3084" (UID: "c502b64e-586e-4b56-9da1-73a4173c3084"). InnerVolumeSpecName "kube-api-access-4tcjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:10.808143 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:10.808114 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c502b64e-586e-4b56-9da1-73a4173c3084-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:10.808143 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:10.808140 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tcjg\" (UniqueName: \"kubernetes.io/projected/c502b64e-586e-4b56-9da1-73a4173c3084-kube-api-access-4tcjg\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:11.502659 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:11.502575 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765" Oct 30 22:48:11.502659 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:11.502626 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765" event={"ID":"c502b64e-586e-4b56-9da1-73a4173c3084","Type":"ContainerDied","Data":"b00bd95b94d2894995ea0c007f29565e9da34adf0551649911b72074ea9844c8"} Oct 30 22:48:11.502825 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:11.502673 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b00bd95b94d2894995ea0c007f29565e9da34adf0551649911b72074ea9844c8" Oct 30 22:48:20.013532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.013501 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf"] Oct 30 22:48:20.015917 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.013980 4335 cpu_manager.go:414] "containerMap: removing container" podUID="c502b64e-586e-4b56-9da1-73a4173c3084" containerName="revoke-keys" Oct 30 22:48:20.015917 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.013994 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c502b64e-586e-4b56-9da1-73a4173c3084" containerName="revoke-keys" Oct 30 22:48:20.015917 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.014067 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="c502b64e-586e-4b56-9da1-73a4173c3084" containerName="revoke-keys" Oct 30 22:48:20.017013 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.016998 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf" Oct 30 22:48:20.070831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.070808 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:48:20.070980 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.070960 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:48:20.091286 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.091253 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd04925b-68af-44c1-b09c-c2c8484a604e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf\" (UID: \"cd04925b-68af-44c1-b09c-c2c8484a604e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf" Oct 30 22:48:20.091431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.091306 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd04925b-68af-44c1-b09c-c2c8484a604e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf\" (UID: \"cd04925b-68af-44c1-b09c-c2c8484a604e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf" Oct 30 22:48:20.091550 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.091534 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lhkg\" (UniqueName: \"kubernetes.io/projected/cd04925b-68af-44c1-b09c-c2c8484a604e-kube-api-access-5lhkg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf\" (UID: \"cd04925b-68af-44c1-b09c-c2c8484a604e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf" Oct 30 22:48:20.120193 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.120166 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:48:20.120324 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.120228 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:48:20.123305 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.123283 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf"] Oct 30 22:48:20.192241 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.192218 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lhkg\" (UniqueName: \"kubernetes.io/projected/cd04925b-68af-44c1-b09c-c2c8484a604e-kube-api-access-5lhkg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf\" (UID: \"cd04925b-68af-44c1-b09c-c2c8484a604e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf" Oct 30 22:48:20.192359 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.192252 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd04925b-68af-44c1-b09c-c2c8484a604e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf\" (UID: \"cd04925b-68af-44c1-b09c-c2c8484a604e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf" Oct 30 22:48:20.192359 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.192273 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd04925b-68af-44c1-b09c-c2c8484a604e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf\" (UID: \"cd04925b-68af-44c1-b09c-c2c8484a604e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf" Oct 30 22:48:20.192958 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.192940 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd04925b-68af-44c1-b09c-c2c8484a604e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf\" (UID: \"cd04925b-68af-44c1-b09c-c2c8484a604e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf" Oct 30 22:48:20.194812 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.194792 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd04925b-68af-44c1-b09c-c2c8484a604e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf\" (UID: \"cd04925b-68af-44c1-b09c-c2c8484a604e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf" Oct 30 22:48:20.252888 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.252866 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lhkg\" (UniqueName: \"kubernetes.io/projected/cd04925b-68af-44c1-b09c-c2c8484a604e-kube-api-access-5lhkg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf\" (UID: \"cd04925b-68af-44c1-b09c-c2c8484a604e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf" Oct 30 22:48:20.330678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.330648 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf" Oct 30 22:48:20.705553 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:20.705525 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf"] Oct 30 22:48:20.706462 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:48:20.706430 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd04925b_68af_44c1_b09c_c2c8484a604e.slice/crio-43cb380a9c0df45f921621514342c36289aaef53d4e458fbd82768864ba95c3c WatchSource:0}: Error finding container 43cb380a9c0df45f921621514342c36289aaef53d4e458fbd82768864ba95c3c: Status 404 returned error can't find the container with id 43cb380a9c0df45f921621514342c36289aaef53d4e458fbd82768864ba95c3c Oct 30 22:48:21.575337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:21.575277 4335 generic.go:356] "Generic (PLEG): container finished" podID="cd04925b-68af-44c1-b09c-c2c8484a604e" containerID="adc49b0997a3ccc41a271dbed589d810f096e7b6232d7a6d10e8250aa2d9596b" exitCode=0 Oct 30 22:48:21.575693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:21.575363 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf" event={"ID":"cd04925b-68af-44c1-b09c-c2c8484a604e","Type":"ContainerDied","Data":"adc49b0997a3ccc41a271dbed589d810f096e7b6232d7a6d10e8250aa2d9596b"} Oct 30 22:48:21.575693 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:21.575430 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf" event={"ID":"cd04925b-68af-44c1-b09c-c2c8484a604e","Type":"ContainerStarted","Data":"43cb380a9c0df45f921621514342c36289aaef53d4e458fbd82768864ba95c3c"} Oct 30 22:48:22.716322 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:22.716296 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf" Oct 30 22:48:22.813533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:22.813503 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd04925b-68af-44c1-b09c-c2c8484a604e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd04925b-68af-44c1-b09c-c2c8484a604e-maas-api-openshift-ca\") pod \"cd04925b-68af-44c1-b09c-c2c8484a604e\" (UID: \"cd04925b-68af-44c1-b09c-c2c8484a604e\") " Oct 30 22:48:22.813678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:22.813560 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd04925b-68af-44c1-b09c-c2c8484a604e-kube-api-access-5lhkg\" (UniqueName: \"kubernetes.io/projected/cd04925b-68af-44c1-b09c-c2c8484a604e-kube-api-access-5lhkg\") pod \"cd04925b-68af-44c1-b09c-c2c8484a604e\" (UID: \"cd04925b-68af-44c1-b09c-c2c8484a604e\") " Oct 30 22:48:22.813678 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:22.813583 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd04925b-68af-44c1-b09c-c2c8484a604e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd04925b-68af-44c1-b09c-c2c8484a604e-maas-api-xks-ca\") pod \"cd04925b-68af-44c1-b09c-c2c8484a604e\" (UID: \"cd04925b-68af-44c1-b09c-c2c8484a604e\") " Oct 30 22:48:22.813846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:22.813826 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd04925b-68af-44c1-b09c-c2c8484a604e-maas-api-openshift-ca" pod "cd04925b-68af-44c1-b09c-c2c8484a604e" (UID: "cd04925b-68af-44c1-b09c-c2c8484a604e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:48:22.814014 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:22.813986 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd04925b-68af-44c1-b09c-c2c8484a604e-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:22.815796 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:22.815769 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd04925b-68af-44c1-b09c-c2c8484a604e-maas-api-xks-ca" pod "cd04925b-68af-44c1-b09c-c2c8484a604e" (UID: "cd04925b-68af-44c1-b09c-c2c8484a604e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:48:22.815879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:22.815793 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd04925b-68af-44c1-b09c-c2c8484a604e-kube-api-access-5lhkg" pod "cd04925b-68af-44c1-b09c-c2c8484a604e" (UID: "cd04925b-68af-44c1-b09c-c2c8484a604e"). InnerVolumeSpecName "kube-api-access-5lhkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:22.844915 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:22.844849 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:48:22.914480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:22.914454 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lhkg\" (UniqueName: \"kubernetes.io/projected/cd04925b-68af-44c1-b09c-c2c8484a604e-kube-api-access-5lhkg\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:22.914480 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:22.914480 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd04925b-68af-44c1-b09c-c2c8484a604e-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:23.589339 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:23.589305 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf" Oct 30 22:48:23.589513 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:23.589312 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf" event={"ID":"cd04925b-68af-44c1-b09c-c2c8484a604e","Type":"ContainerDied","Data":"43cb380a9c0df45f921621514342c36289aaef53d4e458fbd82768864ba95c3c"} Oct 30 22:48:23.589513 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:23.589433 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43cb380a9c0df45f921621514342c36289aaef53d4e458fbd82768864ba95c3c" Oct 30 22:48:24.485346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:24.485302 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh"] Oct 30 22:48:25.320811 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:48:25.320780 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:48:30.511910 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:30.511874 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk"] Oct 30 22:48:30.973407 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:30.973375 4335 scope.go:122] "RemoveContainer" containerID="8149b3126504586428fe7315a5dfe61e0f8a88a1b27b8ae0f95e3e739e0ed9a5" Oct 30 22:48:30.983245 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:30.983221 4335 scope.go:122] "RemoveContainer" containerID="2633d100fb1b03e2c5532b88c2076ea63d676c7925cb4932bf2e2c8ec06a1f47" Oct 30 22:48:32.047370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:32.047340 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47"] Oct 30 22:48:32.050663 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:32.050632 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-btz47"] Oct 30 22:48:33.056202 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:33.056162 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93948790-4aa0-439a-871f-778f6089844e" path="/var/lib/kubelet/pods/93948790-4aa0-439a-871f-778f6089844e/volumes" Oct 30 22:48:35.512939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:35.512900 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" podUID="c95f3db1-460d-4169-845e-af21a6e21e1e" containerName="payload-pre-processing" containerID="cri-o://9a4a3b6e45ff47d2456f97d055c50dde7cb697c36e8b15ed731c5f0566507da1" gracePeriod=25 Oct 30 22:48:35.966197 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:35.966162 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" podUID="c95f3db1-460d-4169-845e-af21a6e21e1e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:48:36.408638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:36.408583 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf"] Oct 30 22:48:36.457537 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:36.457505 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d0a613-mmhqf"] Oct 30 22:48:37.056400 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:37.056370 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd04925b-68af-44c1-b09c-c2c8484a604e" path="/var/lib/kubelet/pods/cd04925b-68af-44c1-b09c-c2c8484a604e/volumes" Oct 30 22:48:38.867284 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:38.867248 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:48:39.505564 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.505514 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" podUID="1f7e91d1-07f1-47b9-ba9a-a9e219957752" containerName="main" containerID="cri-o://deffd2b6abd2299b73772c7b23bf85890f1ee933684673ba6fbc9b993a6f9f19" gracePeriod=45 Oct 30 22:48:39.695811 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.695775 4335 generic.go:356] "Generic (PLEG): container finished" podID="1f7e91d1-07f1-47b9-ba9a-a9e219957752" containerID="deffd2b6abd2299b73772c7b23bf85890f1ee933684673ba6fbc9b993a6f9f19" exitCode=0 Oct 30 22:48:39.695949 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.695841 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" event={"ID":"1f7e91d1-07f1-47b9-ba9a-a9e219957752","Type":"ContainerDied","Data":"deffd2b6abd2299b73772c7b23bf85890f1ee933684673ba6fbc9b993a6f9f19"} Oct 30 22:48:39.759233 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.759176 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:48:39.864929 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.864893 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-home\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-home\") pod \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " Oct 30 22:48:39.865082 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.864951 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f7e91d1-07f1-47b9-ba9a-a9e219957752-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f7e91d1-07f1-47b9-ba9a-a9e219957752-tls-certs\") pod \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " Oct 30 22:48:39.865082 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.865003 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-model-cache\") pod \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " Oct 30 22:48:39.865082 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.865078 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-tmp-dir\") pod \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " Oct 30 22:48:39.865263 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.865122 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-dshm\") pod \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " Oct 30 22:48:39.865263 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.865183 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f7e91d1-07f1-47b9-ba9a-a9e219957752-kube-api-access-slvkl\" (UniqueName: \"kubernetes.io/projected/1f7e91d1-07f1-47b9-ba9a-a9e219957752-kube-api-access-slvkl\") pod \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\" (UID: \"1f7e91d1-07f1-47b9-ba9a-a9e219957752\") " Oct 30 22:48:39.865263 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.865183 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-home" pod "1f7e91d1-07f1-47b9-ba9a-a9e219957752" (UID: "1f7e91d1-07f1-47b9-ba9a-a9e219957752"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:39.865450 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.865261 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-model-cache" pod "1f7e91d1-07f1-47b9-ba9a-a9e219957752" (UID: "1f7e91d1-07f1-47b9-ba9a-a9e219957752"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:39.865736 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.865524 4335 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-home\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:39.865736 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.865546 4335 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-model-cache\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:39.865736 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.865700 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-tmp-dir" pod "1f7e91d1-07f1-47b9-ba9a-a9e219957752" (UID: "1f7e91d1-07f1-47b9-ba9a-a9e219957752"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:39.867434 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.867404 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7e91d1-07f1-47b9-ba9a-a9e219957752-tls-certs" pod "1f7e91d1-07f1-47b9-ba9a-a9e219957752" (UID: "1f7e91d1-07f1-47b9-ba9a-a9e219957752"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:48:39.867808 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.867517 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7e91d1-07f1-47b9-ba9a-a9e219957752-kube-api-access-slvkl" pod "1f7e91d1-07f1-47b9-ba9a-a9e219957752" (UID: "1f7e91d1-07f1-47b9-ba9a-a9e219957752"). InnerVolumeSpecName "kube-api-access-slvkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:39.867808 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.867716 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-dshm" pod "1f7e91d1-07f1-47b9-ba9a-a9e219957752" (UID: "1f7e91d1-07f1-47b9-ba9a-a9e219957752"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:39.966710 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.966678 4335 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f7e91d1-07f1-47b9-ba9a-a9e219957752-tls-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:39.966710 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.966703 4335 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-tmp-dir\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:39.966710 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.966712 4335 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f7e91d1-07f1-47b9-ba9a-a9e219957752-dshm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:39.966940 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:39.966720 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slvkl\" (UniqueName: \"kubernetes.io/projected/1f7e91d1-07f1-47b9-ba9a-a9e219957752-kube-api-access-slvkl\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:40.703383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:40.703347 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" event={"ID":"1f7e91d1-07f1-47b9-ba9a-a9e219957752","Type":"ContainerDied","Data":"e27b5591716440b7d53f3f4b31098709e88f89393e4c4a12dd3bd13bd8155a49"} Oct 30 22:48:40.703555 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:40.703399 4335 scope.go:122] "RemoveContainer" containerID="deffd2b6abd2299b73772c7b23bf85890f1ee933684673ba6fbc9b993a6f9f19" Oct 30 22:48:40.703555 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:40.703400 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh" Oct 30 22:48:40.868351 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:40.868326 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh"] Oct 30 22:48:40.917692 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:40.917663 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-d0a613/simulator-single-main-d0a613-kserve-97c6c7d49-kgzhh"] Oct 30 22:48:41.056980 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:41.056954 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f7e91d1-07f1-47b9-ba9a-a9e219957752" path="/var/lib/kubelet/pods/1f7e91d1-07f1-47b9-ba9a-a9e219957752/volumes" Oct 30 22:48:44.723903 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:44.723864 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67"] Oct 30 22:48:44.772638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:44.772582 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nvm67"] Oct 30 22:48:44.849629 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:44.849586 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht"] Oct 30 22:48:44.850055 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:44.850041 4335 cpu_manager.go:414] "containerMap: removing container" podUID="cd04925b-68af-44c1-b09c-c2c8484a604e" containerName="revoke-keys" Oct 30 22:48:44.850100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:44.850057 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd04925b-68af-44c1-b09c-c2c8484a604e" containerName="revoke-keys" Oct 30 22:48:44.850100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:44.850069 4335 cpu_manager.go:414] "containerMap: removing container" podUID="1f7e91d1-07f1-47b9-ba9a-a9e219957752" containerName="main" Oct 30 22:48:44.850100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:44.850074 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f7e91d1-07f1-47b9-ba9a-a9e219957752" containerName="main" Oct 30 22:48:44.850201 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:44.850138 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd04925b-68af-44c1-b09c-c2c8484a604e" containerName="revoke-keys" Oct 30 22:48:44.850201 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:44.850149 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f7e91d1-07f1-47b9-ba9a-a9e219957752" containerName="main" Oct 30 22:48:44.854580 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:44.854564 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht" Oct 30 22:48:44.908767 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:44.908743 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:48:44.908955 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:44.908794 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:48:44.909470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:44.909247 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v82p5\" (UniqueName: \"kubernetes.io/projected/7481e763-3843-43d2-9731-f5d6c13f62b0-kube-api-access-v82p5\") pod \"maas-api-revoke-sub-3d4156f1-vrwht\" (UID: \"7481e763-3843-43d2-9731-f5d6c13f62b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht" Oct 30 22:48:44.909470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:44.909289 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7481e763-3843-43d2-9731-f5d6c13f62b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vrwht\" (UID: \"7481e763-3843-43d2-9731-f5d6c13f62b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht" Oct 30 22:48:44.909470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:44.909347 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7481e763-3843-43d2-9731-f5d6c13f62b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vrwht\" (UID: \"7481e763-3843-43d2-9731-f5d6c13f62b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht" Oct 30 22:48:44.912746 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:44.912724 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht"] Oct 30 22:48:44.954302 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:44.954279 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:48:44.954412 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:44.954304 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:48:45.010791 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:45.010739 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7481e763-3843-43d2-9731-f5d6c13f62b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vrwht\" (UID: \"7481e763-3843-43d2-9731-f5d6c13f62b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht" Oct 30 22:48:45.010884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:45.010829 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v82p5\" (UniqueName: \"kubernetes.io/projected/7481e763-3843-43d2-9731-f5d6c13f62b0-kube-api-access-v82p5\") pod \"maas-api-revoke-sub-3d4156f1-vrwht\" (UID: \"7481e763-3843-43d2-9731-f5d6c13f62b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht" Oct 30 22:48:45.010884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:45.010856 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7481e763-3843-43d2-9731-f5d6c13f62b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vrwht\" (UID: \"7481e763-3843-43d2-9731-f5d6c13f62b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht" Oct 30 22:48:45.011510 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:45.011485 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7481e763-3843-43d2-9731-f5d6c13f62b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vrwht\" (UID: \"7481e763-3843-43d2-9731-f5d6c13f62b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht" Oct 30 22:48:45.013190 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:45.013168 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7481e763-3843-43d2-9731-f5d6c13f62b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vrwht\" (UID: \"7481e763-3843-43d2-9731-f5d6c13f62b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht" Oct 30 22:48:45.057961 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:45.057936 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4c5709d-563c-48a2-b245-668499bac383" path="/var/lib/kubelet/pods/c4c5709d-563c-48a2-b245-668499bac383/volumes" Oct 30 22:48:45.071632 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:45.071596 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v82p5\" (UniqueName: \"kubernetes.io/projected/7481e763-3843-43d2-9731-f5d6c13f62b0-kube-api-access-v82p5\") pod \"maas-api-revoke-sub-3d4156f1-vrwht\" (UID: \"7481e763-3843-43d2-9731-f5d6c13f62b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht" Oct 30 22:48:45.166755 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:45.166735 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht" Oct 30 22:48:45.553082 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:48:45.553052 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7481e763_3843_43d2_9731_f5d6c13f62b0.slice/crio-748629e7d30b57bd4757f9de6886b7bdbfb92d8f607e80adb4a6d3dffe694f48 WatchSource:0}: Error finding container 748629e7d30b57bd4757f9de6886b7bdbfb92d8f607e80adb4a6d3dffe694f48: Status 404 returned error can't find the container with id 748629e7d30b57bd4757f9de6886b7bdbfb92d8f607e80adb4a6d3dffe694f48 Oct 30 22:48:45.555900 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:45.555863 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht"] Oct 30 22:48:45.738025 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:45.737996 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht" event={"ID":"7481e763-3843-43d2-9731-f5d6c13f62b0","Type":"ContainerStarted","Data":"748629e7d30b57bd4757f9de6886b7bdbfb92d8f607e80adb4a6d3dffe694f48"} Oct 30 22:48:45.965699 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:45.965656 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" podUID="c95f3db1-460d-4169-845e-af21a6e21e1e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:48:46.747198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:46.747115 4335 generic.go:356] "Generic (PLEG): container finished" podID="7481e763-3843-43d2-9731-f5d6c13f62b0" containerID="b4864d058bb4bb08a0b28fb254322823cd4c69c12b3644b464bb8ee3b667fa5e" exitCode=0 Oct 30 22:48:46.747674 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:46.747211 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht" event={"ID":"7481e763-3843-43d2-9731-f5d6c13f62b0","Type":"ContainerDied","Data":"b4864d058bb4bb08a0b28fb254322823cd4c69c12b3644b464bb8ee3b667fa5e"} Oct 30 22:48:47.893554 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:47.893531 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht" Oct 30 22:48:47.940337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:47.940305 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7481e763-3843-43d2-9731-f5d6c13f62b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7481e763-3843-43d2-9731-f5d6c13f62b0-maas-api-xks-ca\") pod \"7481e763-3843-43d2-9731-f5d6c13f62b0\" (UID: \"7481e763-3843-43d2-9731-f5d6c13f62b0\") " Oct 30 22:48:47.940473 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:47.940357 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7481e763-3843-43d2-9731-f5d6c13f62b0-kube-api-access-v82p5\" (UniqueName: \"kubernetes.io/projected/7481e763-3843-43d2-9731-f5d6c13f62b0-kube-api-access-v82p5\") pod \"7481e763-3843-43d2-9731-f5d6c13f62b0\" (UID: \"7481e763-3843-43d2-9731-f5d6c13f62b0\") " Oct 30 22:48:47.940473 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:47.940407 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7481e763-3843-43d2-9731-f5d6c13f62b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7481e763-3843-43d2-9731-f5d6c13f62b0-maas-api-openshift-ca\") pod \"7481e763-3843-43d2-9731-f5d6c13f62b0\" (UID: \"7481e763-3843-43d2-9731-f5d6c13f62b0\") " Oct 30 22:48:47.940830 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:47.940808 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7481e763-3843-43d2-9731-f5d6c13f62b0-maas-api-openshift-ca" pod "7481e763-3843-43d2-9731-f5d6c13f62b0" (UID: "7481e763-3843-43d2-9731-f5d6c13f62b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:48:47.942384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:47.942364 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7481e763-3843-43d2-9731-f5d6c13f62b0-maas-api-xks-ca" pod "7481e763-3843-43d2-9731-f5d6c13f62b0" (UID: "7481e763-3843-43d2-9731-f5d6c13f62b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:48:47.942532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:47.942510 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7481e763-3843-43d2-9731-f5d6c13f62b0-kube-api-access-v82p5" pod "7481e763-3843-43d2-9731-f5d6c13f62b0" (UID: "7481e763-3843-43d2-9731-f5d6c13f62b0"). InnerVolumeSpecName "kube-api-access-v82p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:48.041210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:48.041157 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7481e763-3843-43d2-9731-f5d6c13f62b0-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:48.041210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:48.041181 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v82p5\" (UniqueName: \"kubernetes.io/projected/7481e763-3843-43d2-9731-f5d6c13f62b0-kube-api-access-v82p5\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:48.041210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:48.041191 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7481e763-3843-43d2-9731-f5d6c13f62b0-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:48.761847 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:48.761821 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht" Oct 30 22:48:48.762030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:48.761820 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht" event={"ID":"7481e763-3843-43d2-9731-f5d6c13f62b0","Type":"ContainerDied","Data":"748629e7d30b57bd4757f9de6886b7bdbfb92d8f607e80adb4a6d3dffe694f48"} Oct 30 22:48:48.762030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:48.761938 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="748629e7d30b57bd4757f9de6886b7bdbfb92d8f607e80adb4a6d3dffe694f48" Oct 30 22:48:55.320676 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:48:55.320638 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:48:55.768338 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:55.767995 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" Oct 30 22:48:55.813116 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:55.813088 4335 generic.go:356] "Generic (PLEG): container finished" podID="c95f3db1-460d-4169-845e-af21a6e21e1e" containerID="9a4a3b6e45ff47d2456f97d055c50dde7cb697c36e8b15ed731c5f0566507da1" exitCode=0 Oct 30 22:48:55.813260 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:55.813127 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" Oct 30 22:48:55.813260 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:55.813179 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" event={"ID":"c95f3db1-460d-4169-845e-af21a6e21e1e","Type":"ContainerDied","Data":"9a4a3b6e45ff47d2456f97d055c50dde7cb697c36e8b15ed731c5f0566507da1"} Oct 30 22:48:55.813260 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:55.813236 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk" event={"ID":"c95f3db1-460d-4169-845e-af21a6e21e1e","Type":"ContainerDied","Data":"064849c35de34bfe42cdcea0c92ce643281cccfd7e92c5465be52f33e516603e"} Oct 30 22:48:55.813450 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:55.813264 4335 scope.go:122] "RemoveContainer" containerID="9a4a3b6e45ff47d2456f97d055c50dde7cb697c36e8b15ed731c5f0566507da1" Oct 30 22:48:55.823159 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:55.823141 4335 scope.go:122] "RemoveContainer" containerID="9a4a3b6e45ff47d2456f97d055c50dde7cb697c36e8b15ed731c5f0566507da1" Oct 30 22:48:55.823433 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:48:55.823413 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4a3b6e45ff47d2456f97d055c50dde7cb697c36e8b15ed731c5f0566507da1\": container with ID starting with 9a4a3b6e45ff47d2456f97d055c50dde7cb697c36e8b15ed731c5f0566507da1 not found: ID does not exist" containerID="9a4a3b6e45ff47d2456f97d055c50dde7cb697c36e8b15ed731c5f0566507da1" Oct 30 22:48:55.823478 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:55.823442 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4a3b6e45ff47d2456f97d055c50dde7cb697c36e8b15ed731c5f0566507da1"} err="failed to get container status \"9a4a3b6e45ff47d2456f97d055c50dde7cb697c36e8b15ed731c5f0566507da1\": rpc error: code = NotFound desc = could not find container \"9a4a3b6e45ff47d2456f97d055c50dde7cb697c36e8b15ed731c5f0566507da1\": container with ID starting with 9a4a3b6e45ff47d2456f97d055c50dde7cb697c36e8b15ed731c5f0566507da1 not found: ID does not exist" Oct 30 22:48:55.912884 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:55.912824 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c95f3db1-460d-4169-845e-af21a6e21e1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95f3db1-460d-4169-845e-af21a6e21e1e-plugins-config-volume\") pod \"c95f3db1-460d-4169-845e-af21a6e21e1e\" (UID: \"c95f3db1-460d-4169-845e-af21a6e21e1e\") " Oct 30 22:48:55.913011 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:55.912905 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c95f3db1-460d-4169-845e-af21a6e21e1e-kube-api-access-bkshm\" (UniqueName: \"kubernetes.io/projected/c95f3db1-460d-4169-845e-af21a6e21e1e-kube-api-access-bkshm\") pod \"c95f3db1-460d-4169-845e-af21a6e21e1e\" (UID: \"c95f3db1-460d-4169-845e-af21a6e21e1e\") " Oct 30 22:48:55.913157 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:55.913135 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c95f3db1-460d-4169-845e-af21a6e21e1e-plugins-config-volume" pod "c95f3db1-460d-4169-845e-af21a6e21e1e" (UID: "c95f3db1-460d-4169-845e-af21a6e21e1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:48:55.913225 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:55.913212 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95f3db1-460d-4169-845e-af21a6e21e1e-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:55.914973 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:55.914948 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c95f3db1-460d-4169-845e-af21a6e21e1e-kube-api-access-bkshm" pod "c95f3db1-460d-4169-845e-af21a6e21e1e" (UID: "c95f3db1-460d-4169-845e-af21a6e21e1e"). InnerVolumeSpecName "kube-api-access-bkshm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:56.014508 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:56.014482 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkshm\" (UniqueName: \"kubernetes.io/projected/c95f3db1-460d-4169-845e-af21a6e21e1e-kube-api-access-bkshm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:56.232852 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:56.232786 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk"] Oct 30 22:48:56.238494 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:56.238468 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d0a613-64f5f7t7zk"] Oct 30 22:48:57.062339 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:57.062309 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c95f3db1-460d-4169-845e-af21a6e21e1e" path="/var/lib/kubelet/pods/c95f3db1-460d-4169-845e-af21a6e21e1e/volumes" Oct 30 22:48:59.650538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:48:59.650502 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:49:10.525490 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:10.525457 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:49:19.437173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:19.437140 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx"] Oct 30 22:49:19.437602 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:19.437339 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx" podUID="6928f3c1-1469-4573-a58b-d1436438158c" containerName="manager" containerID="cri-o://ff7a598a04a091a598df5dd1262c36204f93596440b269d01f7fbf81b75d8b6c" gracePeriod=10 Oct 30 22:49:19.689746 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:19.689695 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx" Oct 30 22:49:19.800337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:19.800306 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6928f3c1-1469-4573-a58b-d1436438158c-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6928f3c1-1469-4573-a58b-d1436438158c-extensions-socket-volume\") pod \"6928f3c1-1469-4573-a58b-d1436438158c\" (UID: \"6928f3c1-1469-4573-a58b-d1436438158c\") " Oct 30 22:49:19.800477 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:19.800342 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6928f3c1-1469-4573-a58b-d1436438158c-kube-api-access-dfztw\" (UniqueName: \"kubernetes.io/projected/6928f3c1-1469-4573-a58b-d1436438158c-kube-api-access-dfztw\") pod \"6928f3c1-1469-4573-a58b-d1436438158c\" (UID: \"6928f3c1-1469-4573-a58b-d1436438158c\") " Oct 30 22:49:19.800870 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:19.800835 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6928f3c1-1469-4573-a58b-d1436438158c-extensions-socket-volume" pod "6928f3c1-1469-4573-a58b-d1436438158c" (UID: "6928f3c1-1469-4573-a58b-d1436438158c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:19.802599 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:19.802574 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6928f3c1-1469-4573-a58b-d1436438158c-kube-api-access-dfztw" pod "6928f3c1-1469-4573-a58b-d1436438158c" (UID: "6928f3c1-1469-4573-a58b-d1436438158c"). InnerVolumeSpecName "kube-api-access-dfztw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:19.901318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:19.901291 4335 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6928f3c1-1469-4573-a58b-d1436438158c-extensions-socket-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:49:19.901318 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:19.901319 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfztw\" (UniqueName: \"kubernetes.io/projected/6928f3c1-1469-4573-a58b-d1436438158c-kube-api-access-dfztw\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:49:20.128188 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:20.128148 4335 generic.go:356] "Generic (PLEG): container finished" podID="6928f3c1-1469-4573-a58b-d1436438158c" containerID="ff7a598a04a091a598df5dd1262c36204f93596440b269d01f7fbf81b75d8b6c" exitCode=0 Oct 30 22:49:20.128376 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:20.128192 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx" Oct 30 22:49:20.128376 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:20.128232 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx" event={"ID":"6928f3c1-1469-4573-a58b-d1436438158c","Type":"ContainerDied","Data":"ff7a598a04a091a598df5dd1262c36204f93596440b269d01f7fbf81b75d8b6c"} Oct 30 22:49:20.128376 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:20.128286 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx" event={"ID":"6928f3c1-1469-4573-a58b-d1436438158c","Type":"ContainerDied","Data":"8318451354c601a4db9c68719a22bea539b40fef1926bb1fc22f31476c5a8b0e"} Oct 30 22:49:20.128376 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:20.128308 4335 scope.go:122] "RemoveContainer" containerID="ff7a598a04a091a598df5dd1262c36204f93596440b269d01f7fbf81b75d8b6c" Oct 30 22:49:20.141411 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:20.141396 4335 scope.go:122] "RemoveContainer" containerID="ff7a598a04a091a598df5dd1262c36204f93596440b269d01f7fbf81b75d8b6c" Oct 30 22:49:20.141706 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:49:20.141684 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff7a598a04a091a598df5dd1262c36204f93596440b269d01f7fbf81b75d8b6c\": container with ID starting with ff7a598a04a091a598df5dd1262c36204f93596440b269d01f7fbf81b75d8b6c not found: ID does not exist" containerID="ff7a598a04a091a598df5dd1262c36204f93596440b269d01f7fbf81b75d8b6c" Oct 30 22:49:20.141777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:20.141715 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff7a598a04a091a598df5dd1262c36204f93596440b269d01f7fbf81b75d8b6c"} err="failed to get container status \"ff7a598a04a091a598df5dd1262c36204f93596440b269d01f7fbf81b75d8b6c\": rpc error: code = NotFound desc = could not find container \"ff7a598a04a091a598df5dd1262c36204f93596440b269d01f7fbf81b75d8b6c\": container with ID starting with ff7a598a04a091a598df5dd1262c36204f93596440b269d01f7fbf81b75d8b6c not found: ID does not exist" Oct 30 22:49:20.292074 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:20.292049 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx"] Oct 30 22:49:20.341566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:20.341546 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vk7bx"] Oct 30 22:49:21.056482 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:21.056454 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6928f3c1-1469-4573-a58b-d1436438158c" path="/var/lib/kubelet/pods/6928f3c1-1469-4573-a58b-d1436438158c/volumes" Oct 30 22:49:25.319989 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:49:25.319955 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:49:28.620806 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:28.620773 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:49:28.699871 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:28.699848 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:49:31.009893 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:31.009855 4335 scope.go:122] "RemoveContainer" containerID="ce728c6009548c3c1a6336275a02d4360858b43fe0c866db719f5689d95d2549" Oct 30 22:49:31.020891 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:49:31.020861 4335 scope.go:122] "RemoveContainer" containerID="47d7904d21b99130a90b11b25be1c840aa0f4f2d36594e508ca03c6f8e13fc93" Oct 30 22:49:55.320976 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:49:55.320893 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:50:23.042344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.042311 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsxg"] Oct 30 22:50:23.042814 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.042799 4335 cpu_manager.go:414] "containerMap: removing container" podUID="6928f3c1-1469-4573-a58b-d1436438158c" containerName="manager" Oct 30 22:50:23.042864 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.042815 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6928f3c1-1469-4573-a58b-d1436438158c" containerName="manager" Oct 30 22:50:23.042864 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.042839 4335 cpu_manager.go:414] "containerMap: removing container" podUID="c95f3db1-460d-4169-845e-af21a6e21e1e" containerName="payload-pre-processing" Oct 30 22:50:23.042864 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.042845 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c95f3db1-460d-4169-845e-af21a6e21e1e" containerName="payload-pre-processing" Oct 30 22:50:23.042963 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.042878 4335 cpu_manager.go:414] "containerMap: removing container" podUID="7481e763-3843-43d2-9731-f5d6c13f62b0" containerName="revoke-keys" Oct 30 22:50:23.042963 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.042884 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7481e763-3843-43d2-9731-f5d6c13f62b0" containerName="revoke-keys" Oct 30 22:50:23.042963 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.042956 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="c95f3db1-460d-4169-845e-af21a6e21e1e" containerName="payload-pre-processing" Oct 30 22:50:23.043060 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.042968 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="7481e763-3843-43d2-9731-f5d6c13f62b0" containerName="revoke-keys" Oct 30 22:50:23.043060 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.042977 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="6928f3c1-1469-4573-a58b-d1436438158c" containerName="manager" Oct 30 22:50:23.046247 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.046228 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsxg" Oct 30 22:50:23.102639 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.102600 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsxg"] Oct 30 22:50:23.118306 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.118281 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp25x\" (UniqueName: \"kubernetes.io/projected/8604f22a-5d96-4c12-b812-b9ac529c3dec-kube-api-access-bp25x\") pod \"kuadrant-operator-controller-manager-678d87766f-dwsxg\" (UID: \"8604f22a-5d96-4c12-b812-b9ac529c3dec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsxg" Oct 30 22:50:23.118421 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.118344 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8604f22a-5d96-4c12-b812-b9ac529c3dec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-dwsxg\" (UID: \"8604f22a-5d96-4c12-b812-b9ac529c3dec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsxg" Oct 30 22:50:23.149658 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.149637 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mfkcm\"" Oct 30 22:50:23.219160 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.219132 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8604f22a-5d96-4c12-b812-b9ac529c3dec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-dwsxg\" (UID: \"8604f22a-5d96-4c12-b812-b9ac529c3dec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsxg" Oct 30 22:50:23.219245 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.219226 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp25x\" (UniqueName: \"kubernetes.io/projected/8604f22a-5d96-4c12-b812-b9ac529c3dec-kube-api-access-bp25x\") pod \"kuadrant-operator-controller-manager-678d87766f-dwsxg\" (UID: \"8604f22a-5d96-4c12-b812-b9ac529c3dec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsxg" Oct 30 22:50:23.219552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.219532 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8604f22a-5d96-4c12-b812-b9ac529c3dec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-dwsxg\" (UID: \"8604f22a-5d96-4c12-b812-b9ac529c3dec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsxg" Oct 30 22:50:23.315227 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.315203 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp25x\" (UniqueName: \"kubernetes.io/projected/8604f22a-5d96-4c12-b812-b9ac529c3dec-kube-api-access-bp25x\") pod \"kuadrant-operator-controller-manager-678d87766f-dwsxg\" (UID: \"8604f22a-5d96-4c12-b812-b9ac529c3dec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsxg" Oct 30 22:50:23.359720 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.359693 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsxg" Oct 30 22:50:23.837659 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:50:23.837605 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8604f22a_5d96_4c12_b812_b9ac529c3dec.slice/crio-6d8e588c206d6da8826ae4a44f59179d94dbc74abdff1d889de2dc968499cb98 WatchSource:0}: Error finding container 6d8e588c206d6da8826ae4a44f59179d94dbc74abdff1d889de2dc968499cb98: Status 404 returned error can't find the container with id 6d8e588c206d6da8826ae4a44f59179d94dbc74abdff1d889de2dc968499cb98 Oct 30 22:50:23.885735 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:23.885709 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsxg"] Oct 30 22:50:24.516485 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:24.516454 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsxg" event={"ID":"8604f22a-5d96-4c12-b812-b9ac529c3dec","Type":"ContainerStarted","Data":"c02d038b67f77202e01308c3e9ef5bf6ce548403fc58053ec725a480b0128364"} Oct 30 22:50:24.517029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:24.516994 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsxg" event={"ID":"8604f22a-5d96-4c12-b812-b9ac529c3dec","Type":"ContainerStarted","Data":"6d8e588c206d6da8826ae4a44f59179d94dbc74abdff1d889de2dc968499cb98"} Oct 30 22:50:24.517029 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:24.517032 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsxg" Oct 30 22:50:24.738837 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:24.738786 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsxg" podStartSLOduration=2.738771972 podStartE2EDuration="2.738771972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:24.686852011 +0000 UTC m=+3957.787283742" watchObservedRunningTime="2026-10-30 22:50:24.738771972 +0000 UTC m=+3957.839203697" Oct 30 22:50:25.320182 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:50:25.320150 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:50:29.047795 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:29.047768 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q"] Oct 30 22:50:29.097271 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:29.097243 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b8k8q"] Oct 30 22:50:31.056173 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:31.056141 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e85d04-bd8b-4d53-95c8-acc90ad279c2" path="/var/lib/kubelet/pods/e8e85d04-bd8b-4d53-95c8-acc90ad279c2/volumes" Oct 30 22:50:31.060992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:31.060962 4335 scope.go:122] "RemoveContainer" containerID="abbadd6cc9abb8c210ecbf9134984388165c512a598f5a03403159ab3bc4118f" Oct 30 22:50:31.070106 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:31.070072 4335 scope.go:122] "RemoveContainer" containerID="88bbaf6e730742e0b5f6f69a96452d61314369ba484f01cc33a74a8c06d19383" Oct 30 22:50:36.530163 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:36.530133 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsxg" Oct 30 22:50:40.049275 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:40.049244 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q"] Oct 30 22:50:40.098453 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:40.098425 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7664q"] Oct 30 22:50:41.056112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:41.056082 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="235088f5-1bbc-4d0c-b6af-5d320ad5448c" path="/var/lib/kubelet/pods/235088f5-1bbc-4d0c-b6af-5d320ad5448c/volumes" Oct 30 22:50:46.055634 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:46.055588 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:50:50.653812 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:50.653776 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8"] Oct 30 22:50:50.657549 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:50.657533 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8" Oct 30 22:50:50.683396 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:50.683366 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279a5123-e61b-4ac0-b664-0716a0fc8968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-vdnq8\" (UID: \"279a5123-e61b-4ac0-b664-0716a0fc8968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8" Oct 30 22:50:50.683508 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:50.683434 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9w2z\" (UniqueName: \"kubernetes.io/projected/279a5123-e61b-4ac0-b664-0716a0fc8968-kube-api-access-j9w2z\") pod \"maas-api-revoke-sub-d576e68b-vdnq8\" (UID: \"279a5123-e61b-4ac0-b664-0716a0fc8968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8" Oct 30 22:50:50.683563 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:50.683529 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279a5123-e61b-4ac0-b664-0716a0fc8968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-vdnq8\" (UID: \"279a5123-e61b-4ac0-b664-0716a0fc8968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8" Oct 30 22:50:50.761486 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:50.761454 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:50:50.761661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:50.761555 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:50:50.761661 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:50.761572 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:50:50.761849 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:50.761835 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:50:50.764852 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:50.764834 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8"] Oct 30 22:50:50.784109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:50.784086 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9w2z\" (UniqueName: \"kubernetes.io/projected/279a5123-e61b-4ac0-b664-0716a0fc8968-kube-api-access-j9w2z\") pod \"maas-api-revoke-sub-d576e68b-vdnq8\" (UID: \"279a5123-e61b-4ac0-b664-0716a0fc8968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8" Oct 30 22:50:50.784219 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:50.784134 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279a5123-e61b-4ac0-b664-0716a0fc8968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-vdnq8\" (UID: \"279a5123-e61b-4ac0-b664-0716a0fc8968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8" Oct 30 22:50:50.784219 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:50.784191 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279a5123-e61b-4ac0-b664-0716a0fc8968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-vdnq8\" (UID: \"279a5123-e61b-4ac0-b664-0716a0fc8968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8" Oct 30 22:50:50.784767 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:50.784746 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279a5123-e61b-4ac0-b664-0716a0fc8968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-vdnq8\" (UID: \"279a5123-e61b-4ac0-b664-0716a0fc8968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8" Oct 30 22:50:50.786660 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:50.786642 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279a5123-e61b-4ac0-b664-0716a0fc8968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-vdnq8\" (UID: \"279a5123-e61b-4ac0-b664-0716a0fc8968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8" Oct 30 22:50:50.844892 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:50.844863 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9w2z\" (UniqueName: \"kubernetes.io/projected/279a5123-e61b-4ac0-b664-0716a0fc8968-kube-api-access-j9w2z\") pod \"maas-api-revoke-sub-d576e68b-vdnq8\" (UID: \"279a5123-e61b-4ac0-b664-0716a0fc8968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8" Oct 30 22:50:50.970257 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:50.970195 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8" Oct 30 22:50:51.346677 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:51.346651 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8"] Oct 30 22:50:51.348430 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:50:51.348404 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279a5123_e61b_4ac0_b664_0716a0fc8968.slice/crio-75ffc267ab2ea14666dfd75b832de258dfe0e03d16540ecada07291b061fc1cf WatchSource:0}: Error finding container 75ffc267ab2ea14666dfd75b832de258dfe0e03d16540ecada07291b061fc1cf: Status 404 returned error can't find the container with id 75ffc267ab2ea14666dfd75b832de258dfe0e03d16540ecada07291b061fc1cf Oct 30 22:50:51.764445 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:51.764409 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8" event={"ID":"279a5123-e61b-4ac0-b664-0716a0fc8968","Type":"ContainerStarted","Data":"4a198847a050ba2a2c6dffd8eb5a2398261b2c53e39ee78e86fa8d5a5f93277b"} Oct 30 22:50:51.764868 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:51.764473 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8" event={"ID":"279a5123-e61b-4ac0-b664-0716a0fc8968","Type":"ContainerStarted","Data":"75ffc267ab2ea14666dfd75b832de258dfe0e03d16540ecada07291b061fc1cf"} Oct 30 22:50:51.881060 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:51.880973 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8" podStartSLOduration=1.880954838 podStartE2EDuration="1.880954838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:51.87909319 +0000 UTC m=+3984.979524921" watchObservedRunningTime="2026-10-30 22:50:51.880954838 +0000 UTC m=+3984.981386570" Oct 30 22:50:52.773454 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:52.773417 4335 generic.go:356] "Generic (PLEG): container finished" podID="279a5123-e61b-4ac0-b664-0716a0fc8968" containerID="4a198847a050ba2a2c6dffd8eb5a2398261b2c53e39ee78e86fa8d5a5f93277b" exitCode=0 Oct 30 22:50:52.773928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:52.773498 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8" event={"ID":"279a5123-e61b-4ac0-b664-0716a0fc8968","Type":"ContainerDied","Data":"4a198847a050ba2a2c6dffd8eb5a2398261b2c53e39ee78e86fa8d5a5f93277b"} Oct 30 22:50:53.916604 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:53.916581 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8" Oct 30 22:50:54.008685 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:54.008652 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279a5123-e61b-4ac0-b664-0716a0fc8968-kube-api-access-j9w2z\" (UniqueName: \"kubernetes.io/projected/279a5123-e61b-4ac0-b664-0716a0fc8968-kube-api-access-j9w2z\") pod \"279a5123-e61b-4ac0-b664-0716a0fc8968\" (UID: \"279a5123-e61b-4ac0-b664-0716a0fc8968\") " Oct 30 22:50:54.008816 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:54.008802 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/279a5123-e61b-4ac0-b664-0716a0fc8968-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279a5123-e61b-4ac0-b664-0716a0fc8968-maas-api-xks-ca\") pod \"279a5123-e61b-4ac0-b664-0716a0fc8968\" (UID: \"279a5123-e61b-4ac0-b664-0716a0fc8968\") " Oct 30 22:50:54.008867 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:54.008827 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/279a5123-e61b-4ac0-b664-0716a0fc8968-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279a5123-e61b-4ac0-b664-0716a0fc8968-maas-api-openshift-ca\") pod \"279a5123-e61b-4ac0-b664-0716a0fc8968\" (UID: \"279a5123-e61b-4ac0-b664-0716a0fc8968\") " Oct 30 22:50:54.009209 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:54.009186 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279a5123-e61b-4ac0-b664-0716a0fc8968-maas-api-openshift-ca" pod "279a5123-e61b-4ac0-b664-0716a0fc8968" (UID: "279a5123-e61b-4ac0-b664-0716a0fc8968"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:50:54.010825 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:54.010804 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279a5123-e61b-4ac0-b664-0716a0fc8968-kube-api-access-j9w2z" pod "279a5123-e61b-4ac0-b664-0716a0fc8968" (UID: "279a5123-e61b-4ac0-b664-0716a0fc8968"). InnerVolumeSpecName "kube-api-access-j9w2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:50:54.010919 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:54.010871 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/279a5123-e61b-4ac0-b664-0716a0fc8968-maas-api-xks-ca" pod "279a5123-e61b-4ac0-b664-0716a0fc8968" (UID: "279a5123-e61b-4ac0-b664-0716a0fc8968"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:50:54.110307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:54.110279 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279a5123-e61b-4ac0-b664-0716a0fc8968-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:54.110307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:54.110306 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279a5123-e61b-4ac0-b664-0716a0fc8968-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:54.110470 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:54.110316 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9w2z\" (UniqueName: \"kubernetes.io/projected/279a5123-e61b-4ac0-b664-0716a0fc8968-kube-api-access-j9w2z\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:50:54.787354 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:54.787322 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8" Oct 30 22:50:54.787354 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:54.787336 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8" event={"ID":"279a5123-e61b-4ac0-b664-0716a0fc8968","Type":"ContainerDied","Data":"75ffc267ab2ea14666dfd75b832de258dfe0e03d16540ecada07291b061fc1cf"} Oct 30 22:50:54.787579 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:50:54.787393 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75ffc267ab2ea14666dfd75b832de258dfe0e03d16540ecada07291b061fc1cf" Oct 30 22:50:55.319245 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:50:55.319210 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:51:03.629046 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:03.629006 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:51:11.058810 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:11.058722 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-mjgkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:51:11.059388 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:11.059355 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" podUID="4ed2ae60-d2b4-4ce7-b48b-295a7eabe663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:51:13.443346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:13.443311 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:51:19.898343 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:19.898311 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q"] Oct 30 22:51:19.898977 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:19.898956 4335 cpu_manager.go:414] "containerMap: removing container" podUID="279a5123-e61b-4ac0-b664-0716a0fc8968" containerName="revoke-keys" Oct 30 22:51:19.899037 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:19.898980 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279a5123-e61b-4ac0-b664-0716a0fc8968" containerName="revoke-keys" Oct 30 22:51:19.899100 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:19.899089 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="279a5123-e61b-4ac0-b664-0716a0fc8968" containerName="revoke-keys" Oct 30 22:51:19.902662 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:19.902643 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q" Oct 30 22:51:19.965355 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:19.965307 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949ecb4d-efd6-4b82-af45-56f8d7e33d00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-s5n8q\" (UID: \"949ecb4d-efd6-4b82-af45-56f8d7e33d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q" Oct 30 22:51:19.965688 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:19.965669 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949ecb4d-efd6-4b82-af45-56f8d7e33d00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-s5n8q\" (UID: \"949ecb4d-efd6-4b82-af45-56f8d7e33d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q" Oct 30 22:51:19.965908 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:19.965892 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc5kr\" (UniqueName: \"kubernetes.io/projected/949ecb4d-efd6-4b82-af45-56f8d7e33d00-kube-api-access-pc5kr\") pod \"maas-api-revoke-sub-d2bc93bc-s5n8q\" (UID: \"949ecb4d-efd6-4b82-af45-56f8d7e33d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q" Oct 30 22:51:20.004327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:20.004294 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:51:20.004327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:20.004294 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:51:20.004327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:20.004297 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:51:20.004527 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:20.004304 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:51:20.007456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:20.007434 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q"] Oct 30 22:51:20.067092 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:20.067068 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949ecb4d-efd6-4b82-af45-56f8d7e33d00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-s5n8q\" (UID: \"949ecb4d-efd6-4b82-af45-56f8d7e33d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q" Oct 30 22:51:20.067244 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:20.067138 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc5kr\" (UniqueName: \"kubernetes.io/projected/949ecb4d-efd6-4b82-af45-56f8d7e33d00-kube-api-access-pc5kr\") pod \"maas-api-revoke-sub-d2bc93bc-s5n8q\" (UID: \"949ecb4d-efd6-4b82-af45-56f8d7e33d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q" Oct 30 22:51:20.067244 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:20.067204 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949ecb4d-efd6-4b82-af45-56f8d7e33d00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-s5n8q\" (UID: \"949ecb4d-efd6-4b82-af45-56f8d7e33d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q" Oct 30 22:51:20.067907 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:20.067886 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949ecb4d-efd6-4b82-af45-56f8d7e33d00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-s5n8q\" (UID: \"949ecb4d-efd6-4b82-af45-56f8d7e33d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q" Oct 30 22:51:20.069638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:20.069600 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949ecb4d-efd6-4b82-af45-56f8d7e33d00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-s5n8q\" (UID: \"949ecb4d-efd6-4b82-af45-56f8d7e33d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q" Oct 30 22:51:20.128325 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:20.128298 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc5kr\" (UniqueName: \"kubernetes.io/projected/949ecb4d-efd6-4b82-af45-56f8d7e33d00-kube-api-access-pc5kr\") pod \"maas-api-revoke-sub-d2bc93bc-s5n8q\" (UID: \"949ecb4d-efd6-4b82-af45-56f8d7e33d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q" Oct 30 22:51:20.219098 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:20.219051 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q" Oct 30 22:51:20.593341 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:20.593314 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q"] Oct 30 22:51:20.594660 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:51:20.594634 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949ecb4d_efd6_4b82_af45_56f8d7e33d00.slice/crio-154f2ca3d546c9c468ac2da97dc8b95f28495dc98875b5a5148495866e603757 WatchSource:0}: Error finding container 154f2ca3d546c9c468ac2da97dc8b95f28495dc98875b5a5148495866e603757: Status 404 returned error can't find the container with id 154f2ca3d546c9c468ac2da97dc8b95f28495dc98875b5a5148495866e603757 Oct 30 22:51:21.042008 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:21.041979 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q" event={"ID":"949ecb4d-efd6-4b82-af45-56f8d7e33d00","Type":"ContainerStarted","Data":"1b6c3bf0b03ee4f281ccec0fa70d6ac66deb4d99eb9cf972b2baee0455ccac1d"} Oct 30 22:51:21.042327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:21.042038 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q" event={"ID":"949ecb4d-efd6-4b82-af45-56f8d7e33d00","Type":"ContainerStarted","Data":"154f2ca3d546c9c468ac2da97dc8b95f28495dc98875b5a5148495866e603757"} Oct 30 22:51:21.156873 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:21.156666 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q" podStartSLOduration=2.156652112 podStartE2EDuration="2.156652112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:21.156444499 +0000 UTC m=+4014.256876266" watchObservedRunningTime="2026-10-30 22:51:21.156652112 +0000 UTC m=+4014.257083846" Oct 30 22:51:22.049647 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:22.049605 4335 generic.go:356] "Generic (PLEG): container finished" podID="949ecb4d-efd6-4b82-af45-56f8d7e33d00" containerID="1b6c3bf0b03ee4f281ccec0fa70d6ac66deb4d99eb9cf972b2baee0455ccac1d" exitCode=0 Oct 30 22:51:22.050023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:22.049707 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q" event={"ID":"949ecb4d-efd6-4b82-af45-56f8d7e33d00","Type":"ContainerDied","Data":"1b6c3bf0b03ee4f281ccec0fa70d6ac66deb4d99eb9cf972b2baee0455ccac1d"} Oct 30 22:51:23.193175 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:23.193149 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q" Oct 30 22:51:23.295926 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:23.295902 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/949ecb4d-efd6-4b82-af45-56f8d7e33d00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949ecb4d-efd6-4b82-af45-56f8d7e33d00-maas-api-openshift-ca\") pod \"949ecb4d-efd6-4b82-af45-56f8d7e33d00\" (UID: \"949ecb4d-efd6-4b82-af45-56f8d7e33d00\") " Oct 30 22:51:23.296053 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:23.295977 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/949ecb4d-efd6-4b82-af45-56f8d7e33d00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949ecb4d-efd6-4b82-af45-56f8d7e33d00-maas-api-xks-ca\") pod \"949ecb4d-efd6-4b82-af45-56f8d7e33d00\" (UID: \"949ecb4d-efd6-4b82-af45-56f8d7e33d00\") " Oct 30 22:51:23.296124 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:23.296055 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/949ecb4d-efd6-4b82-af45-56f8d7e33d00-kube-api-access-pc5kr\" (UniqueName: \"kubernetes.io/projected/949ecb4d-efd6-4b82-af45-56f8d7e33d00-kube-api-access-pc5kr\") pod \"949ecb4d-efd6-4b82-af45-56f8d7e33d00\" (UID: \"949ecb4d-efd6-4b82-af45-56f8d7e33d00\") " Oct 30 22:51:23.296270 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:23.296243 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949ecb4d-efd6-4b82-af45-56f8d7e33d00-maas-api-openshift-ca" pod "949ecb4d-efd6-4b82-af45-56f8d7e33d00" (UID: "949ecb4d-efd6-4b82-af45-56f8d7e33d00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:23.298225 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:23.298201 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/949ecb4d-efd6-4b82-af45-56f8d7e33d00-maas-api-xks-ca" pod "949ecb4d-efd6-4b82-af45-56f8d7e33d00" (UID: "949ecb4d-efd6-4b82-af45-56f8d7e33d00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:51:23.298312 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:23.298266 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949ecb4d-efd6-4b82-af45-56f8d7e33d00-kube-api-access-pc5kr" pod "949ecb4d-efd6-4b82-af45-56f8d7e33d00" (UID: "949ecb4d-efd6-4b82-af45-56f8d7e33d00"). InnerVolumeSpecName "kube-api-access-pc5kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:23.397577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:23.397524 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949ecb4d-efd6-4b82-af45-56f8d7e33d00-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:23.397577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:23.397547 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc5kr\" (UniqueName: \"kubernetes.io/projected/949ecb4d-efd6-4b82-af45-56f8d7e33d00-kube-api-access-pc5kr\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:23.397577 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:23.397558 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949ecb4d-efd6-4b82-af45-56f8d7e33d00-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:24.063294 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:24.063272 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q" Oct 30 22:51:24.063441 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:24.063312 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q" event={"ID":"949ecb4d-efd6-4b82-af45-56f8d7e33d00","Type":"ContainerDied","Data":"154f2ca3d546c9c468ac2da97dc8b95f28495dc98875b5a5148495866e603757"} Oct 30 22:51:24.063441 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:24.063345 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="154f2ca3d546c9c468ac2da97dc8b95f28495dc98875b5a5148495866e603757" Oct 30 22:51:25.320770 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:51:25.320734 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:51:31.095190 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:31.095154 4335 scope.go:122] "RemoveContainer" containerID="04f3f8d8720f16c46b3179c5d02c4a56d386d06ebb07a5c92a95639e1735f034" Oct 30 22:51:32.852967 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:32.852935 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:51:40.837190 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:40.837145 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d"] Oct 30 22:51:40.837701 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:40.837566 4335 cpu_manager.go:414] "containerMap: removing container" podUID="949ecb4d-efd6-4b82-af45-56f8d7e33d00" containerName="revoke-keys" Oct 30 22:51:40.837701 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:40.837576 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="949ecb4d-efd6-4b82-af45-56f8d7e33d00" containerName="revoke-keys" Oct 30 22:51:40.837701 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:40.837672 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="949ecb4d-efd6-4b82-af45-56f8d7e33d00" containerName="revoke-keys" Oct 30 22:51:40.840831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:40.840815 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" Oct 30 22:51:40.898882 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:40.898855 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d"] Oct 30 22:51:40.944491 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:40.944459 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-bf99ca\"" Oct 30 22:51:40.951003 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:40.950974 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcttq\" (UniqueName: \"kubernetes.io/projected/9ea43e1e-de80-4d46-bf75-e3e19bd17a5c-kube-api-access-mcttq\") pod \"payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d\" (UID: \"9ea43e1e-de80-4d46-bf75-e3e19bd17a5c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" Oct 30 22:51:40.951139 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:40.951122 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea43e1e-de80-4d46-bf75-e3e19bd17a5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d\" (UID: \"9ea43e1e-de80-4d46-bf75-e3e19bd17a5c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" Oct 30 22:51:41.052655 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:41.052599 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea43e1e-de80-4d46-bf75-e3e19bd17a5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d\" (UID: \"9ea43e1e-de80-4d46-bf75-e3e19bd17a5c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" Oct 30 22:51:41.052858 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:41.052694 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcttq\" (UniqueName: \"kubernetes.io/projected/9ea43e1e-de80-4d46-bf75-e3e19bd17a5c-kube-api-access-mcttq\") pod \"payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d\" (UID: \"9ea43e1e-de80-4d46-bf75-e3e19bd17a5c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" Oct 30 22:51:41.053939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:41.053895 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea43e1e-de80-4d46-bf75-e3e19bd17a5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d\" (UID: \"9ea43e1e-de80-4d46-bf75-e3e19bd17a5c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" Oct 30 22:51:41.113696 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:41.113638 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcttq\" (UniqueName: \"kubernetes.io/projected/9ea43e1e-de80-4d46-bf75-e3e19bd17a5c-kube-api-access-mcttq\") pod \"payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d\" (UID: \"9ea43e1e-de80-4d46-bf75-e3e19bd17a5c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" Oct 30 22:51:41.154566 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:41.154539 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" Oct 30 22:51:41.530809 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:41.530781 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d"] Oct 30 22:51:41.531794 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:51:41.531768 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea43e1e_de80_4d46_bf75_e3e19bd17a5c.slice/crio-004d03f76fac0009b70adbab83839403e77f89f54a92676afaf164c4d12e1604 WatchSource:0}: Error finding container 004d03f76fac0009b70adbab83839403e77f89f54a92676afaf164c4d12e1604: Status 404 returned error can't find the container with id 004d03f76fac0009b70adbab83839403e77f89f54a92676afaf164c4d12e1604 Oct 30 22:51:42.181192 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:42.181160 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" event={"ID":"9ea43e1e-de80-4d46-bf75-e3e19bd17a5c","Type":"ContainerStarted","Data":"67bad79966aaee507861ebf0fc3c94d734b1fbc11d7c751207c33c71d28cd0f0"} Oct 30 22:51:42.181563 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:42.181213 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" event={"ID":"9ea43e1e-de80-4d46-bf75-e3e19bd17a5c","Type":"ContainerStarted","Data":"004d03f76fac0009b70adbab83839403e77f89f54a92676afaf164c4d12e1604"} Oct 30 22:51:42.181563 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:42.181238 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" Oct 30 22:51:42.299412 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:42.299356 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" podStartSLOduration=2.299339092 podStartE2EDuration="2.299339092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:42.295796001 +0000 UTC m=+4035.396227736" watchObservedRunningTime="2026-10-30 22:51:42.299339092 +0000 UTC m=+4035.399770824" Oct 30 22:51:51.620421 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:51.620386 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:51:53.189632 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:51:53.189589 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" Oct 30 22:51:55.320375 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:51:55.320341 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:52:06.044978 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.044939 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6"] Oct 30 22:52:06.048426 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.048409 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.066813 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.066777 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e07a315-b421-4dd2-8d3b-6d241c212377-tls-certs\") pod \"simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.066959 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.066838 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-home\") pod \"simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.066959 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.066900 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-dshm\") pod \"simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.067228 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.067183 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-tmp-dir\") pod \"simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.067327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.067261 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-model-cache\") pod \"simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.067419 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.067390 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7m6v\" (UniqueName: \"kubernetes.io/projected/5e07a315-b421-4dd2-8d3b-6d241c212377-kube-api-access-j7m6v\") pod \"simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.115777 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.115744 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6"] Oct 30 22:52:06.153247 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.153217 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-bf99ca\"/\"kube-root-ca.crt\"" Oct 30 22:52:06.153369 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.153216 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-bf99ca\"/\"openshift-service-ca.crt\"" Oct 30 22:52:06.153369 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.153227 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-bf99ca\"/\"default-dockercfg-znrfx\"" Oct 30 22:52:06.153369 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.153299 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-bf99ca\"/\"simulator-single-main-bf99ca-kserve-self-signed-certs\"" Oct 30 22:52:06.168249 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.168224 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-tmp-dir\") pod \"simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.168369 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.168252 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-model-cache\") pod \"simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.168369 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.168297 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7m6v\" (UniqueName: \"kubernetes.io/projected/5e07a315-b421-4dd2-8d3b-6d241c212377-kube-api-access-j7m6v\") pod \"simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.168369 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.168351 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e07a315-b421-4dd2-8d3b-6d241c212377-tls-certs\") pod \"simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.168554 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.168392 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-home\") pod \"simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.168554 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.168431 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-dshm\") pod \"simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.168744 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.168726 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-tmp-dir\") pod \"simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.168806 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.168750 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-model-cache\") pod \"simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.168806 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.168801 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-home\") pod \"simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.170759 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.170739 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-dshm\") pod \"simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.171009 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.170991 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e07a315-b421-4dd2-8d3b-6d241c212377-tls-certs\") pod \"simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.229180 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.229146 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7m6v\" (UniqueName: \"kubernetes.io/projected/5e07a315-b421-4dd2-8d3b-6d241c212377-kube-api-access-j7m6v\") pod \"simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.360190 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.360163 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:06.738192 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:06.738168 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6"] Oct 30 22:52:06.738999 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:52:06.738971 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e07a315_b421_4dd2_8d3b_6d241c212377.slice/crio-20877325e1c3f802018e271b3986eaec2a35dce44a8047677738ae16a5ce62e6 WatchSource:0}: Error finding container 20877325e1c3f802018e271b3986eaec2a35dce44a8047677738ae16a5ce62e6: Status 404 returned error can't find the container with id 20877325e1c3f802018e271b3986eaec2a35dce44a8047677738ae16a5ce62e6 Oct 30 22:52:07.333941 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:07.333892 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" event={"ID":"5e07a315-b421-4dd2-8d3b-6d241c212377","Type":"ContainerStarted","Data":"8ef46be4bdb499758d200ff6a921e75e47ff6e8360b518ee18c18d71a98cd710"} Oct 30 22:52:07.334482 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:07.333951 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" event={"ID":"5e07a315-b421-4dd2-8d3b-6d241c212377","Type":"ContainerStarted","Data":"20877325e1c3f802018e271b3986eaec2a35dce44a8047677738ae16a5ce62e6"} Oct 30 22:52:07.360327 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:07.360300 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:07.451416 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:07.451375 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" podStartSLOduration=2.451363481 podStartE2EDuration="2.451363481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:07.449956006 +0000 UTC m=+4060.550387761" watchObservedRunningTime="2026-10-30 22:52:07.451363481 +0000 UTC m=+4060.551795212" Oct 30 22:52:10.760361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:10.760327 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:52:16.360340 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:16.360307 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:16.373303 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:16.373280 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:16.402676 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:16.402649 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:52:25.320454 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:52:25.320416 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:52:31.122252 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:31.122217 4335 scope.go:122] "RemoveContainer" containerID="3ea3ceef4284bc444cca11b0394e750ae01640274367dd25dcb5c48bd73fb772" Oct 30 22:52:31.132372 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:31.132352 4335 scope.go:122] "RemoveContainer" containerID="3ae58b923070faf0de44ce0c13ebc526169d056335f37878ab6e6ac19964165b" Oct 30 22:52:36.031633 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:36.031578 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:52:50.995448 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:50.995363 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt"] Oct 30 22:52:50.999339 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:50.999316 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt" Oct 30 22:52:51.052986 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:51.052805 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:52:51.058096 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:51.058071 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt"] Oct 30 22:52:51.102664 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:51.102641 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:52:51.102776 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:51.102681 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:52:51.102776 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:51.102734 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:52:51.159911 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:51.159886 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf5288d-d3d4-4613-96cc-f12f166f4112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f99a8e69-q5jvt\" (UID: \"edf5288d-d3d4-4613-96cc-f12f166f4112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt" Oct 30 22:52:51.160016 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:51.159997 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf5288d-d3d4-4613-96cc-f12f166f4112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f99a8e69-q5jvt\" (UID: \"edf5288d-d3d4-4613-96cc-f12f166f4112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt" Oct 30 22:52:51.160059 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:51.160032 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzzkb\" (UniqueName: \"kubernetes.io/projected/edf5288d-d3d4-4613-96cc-f12f166f4112-kube-api-access-wzzkb\") pod \"maas-api-revoke-sub-f99a8e69-q5jvt\" (UID: \"edf5288d-d3d4-4613-96cc-f12f166f4112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt" Oct 30 22:52:51.260819 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:51.260757 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf5288d-d3d4-4613-96cc-f12f166f4112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f99a8e69-q5jvt\" (UID: \"edf5288d-d3d4-4613-96cc-f12f166f4112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt" Oct 30 22:52:51.260819 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:51.260809 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzzkb\" (UniqueName: \"kubernetes.io/projected/edf5288d-d3d4-4613-96cc-f12f166f4112-kube-api-access-wzzkb\") pod \"maas-api-revoke-sub-f99a8e69-q5jvt\" (UID: \"edf5288d-d3d4-4613-96cc-f12f166f4112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt" Oct 30 22:52:51.261007 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:51.260875 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf5288d-d3d4-4613-96cc-f12f166f4112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f99a8e69-q5jvt\" (UID: \"edf5288d-d3d4-4613-96cc-f12f166f4112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt" Oct 30 22:52:51.261483 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:51.261466 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf5288d-d3d4-4613-96cc-f12f166f4112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f99a8e69-q5jvt\" (UID: \"edf5288d-d3d4-4613-96cc-f12f166f4112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt" Oct 30 22:52:51.263225 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:51.263196 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf5288d-d3d4-4613-96cc-f12f166f4112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f99a8e69-q5jvt\" (UID: \"edf5288d-d3d4-4613-96cc-f12f166f4112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt" Oct 30 22:52:51.322805 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:51.322780 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzzkb\" (UniqueName: \"kubernetes.io/projected/edf5288d-d3d4-4613-96cc-f12f166f4112-kube-api-access-wzzkb\") pod \"maas-api-revoke-sub-f99a8e69-q5jvt\" (UID: \"edf5288d-d3d4-4613-96cc-f12f166f4112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt" Oct 30 22:52:51.611892 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:51.611851 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt" Oct 30 22:52:51.993198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:51.993174 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt"] Oct 30 22:52:51.994209 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:52:51.994175 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf5288d_d3d4_4613_96cc_f12f166f4112.slice/crio-b48f6c24b56442b4c26e5795d2c655d605e075491d914fa85fa84ac4718c1085 WatchSource:0}: Error finding container b48f6c24b56442b4c26e5795d2c655d605e075491d914fa85fa84ac4718c1085: Status 404 returned error can't find the container with id b48f6c24b56442b4c26e5795d2c655d605e075491d914fa85fa84ac4718c1085 Oct 30 22:52:52.612382 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:52.612286 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt" event={"ID":"edf5288d-d3d4-4613-96cc-f12f166f4112","Type":"ContainerStarted","Data":"4774c5ba9cfc8e40d2a94793073ee971fbb530a8554654f0233613b96fce4261"} Oct 30 22:52:52.612382 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:52.612356 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt" event={"ID":"edf5288d-d3d4-4613-96cc-f12f166f4112","Type":"ContainerStarted","Data":"b48f6c24b56442b4c26e5795d2c655d605e075491d914fa85fa84ac4718c1085"} Oct 30 22:52:52.729244 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:52.729180 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt" podStartSLOduration=2.729164132 podStartE2EDuration="2.729164132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:52.72825388 +0000 UTC m=+4105.828685614" watchObservedRunningTime="2026-10-30 22:52:52.729164132 +0000 UTC m=+4105.829595865" Oct 30 22:52:53.620012 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:53.619974 4335 generic.go:356] "Generic (PLEG): container finished" podID="edf5288d-d3d4-4613-96cc-f12f166f4112" containerID="4774c5ba9cfc8e40d2a94793073ee971fbb530a8554654f0233613b96fce4261" exitCode=0 Oct 30 22:52:53.620426 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:53.620064 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt" event={"ID":"edf5288d-d3d4-4613-96cc-f12f166f4112","Type":"ContainerDied","Data":"4774c5ba9cfc8e40d2a94793073ee971fbb530a8554654f0233613b96fce4261"} Oct 30 22:52:54.764034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:54.764009 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt" Oct 30 22:52:54.895636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:54.895552 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf5288d-d3d4-4613-96cc-f12f166f4112-kube-api-access-wzzkb\" (UniqueName: \"kubernetes.io/projected/edf5288d-d3d4-4613-96cc-f12f166f4112-kube-api-access-wzzkb\") pod \"edf5288d-d3d4-4613-96cc-f12f166f4112\" (UID: \"edf5288d-d3d4-4613-96cc-f12f166f4112\") " Oct 30 22:52:54.895636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:54.895591 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edf5288d-d3d4-4613-96cc-f12f166f4112-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf5288d-d3d4-4613-96cc-f12f166f4112-maas-api-openshift-ca\") pod \"edf5288d-d3d4-4613-96cc-f12f166f4112\" (UID: \"edf5288d-d3d4-4613-96cc-f12f166f4112\") " Oct 30 22:52:54.895826 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:54.895741 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edf5288d-d3d4-4613-96cc-f12f166f4112-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf5288d-d3d4-4613-96cc-f12f166f4112-maas-api-xks-ca\") pod \"edf5288d-d3d4-4613-96cc-f12f166f4112\" (UID: \"edf5288d-d3d4-4613-96cc-f12f166f4112\") " Oct 30 22:52:54.895976 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:54.895953 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf5288d-d3d4-4613-96cc-f12f166f4112-maas-api-openshift-ca" pod "edf5288d-d3d4-4613-96cc-f12f166f4112" (UID: "edf5288d-d3d4-4613-96cc-f12f166f4112"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:54.896127 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:54.896112 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf5288d-d3d4-4613-96cc-f12f166f4112-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:54.898037 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:54.898005 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf5288d-d3d4-4613-96cc-f12f166f4112-maas-api-xks-ca" pod "edf5288d-d3d4-4613-96cc-f12f166f4112" (UID: "edf5288d-d3d4-4613-96cc-f12f166f4112"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:54.898169 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:54.898106 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf5288d-d3d4-4613-96cc-f12f166f4112-kube-api-access-wzzkb" pod "edf5288d-d3d4-4613-96cc-f12f166f4112" (UID: "edf5288d-d3d4-4613-96cc-f12f166f4112"). InnerVolumeSpecName "kube-api-access-wzzkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:54.997041 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:54.997007 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzzkb\" (UniqueName: \"kubernetes.io/projected/edf5288d-d3d4-4613-96cc-f12f166f4112-kube-api-access-wzzkb\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:54.997041 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:54.997035 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf5288d-d3d4-4613-96cc-f12f166f4112-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:52:55.320329 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:52:55.320297 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:52:55.632701 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:55.632630 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt" Oct 30 22:52:55.632701 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:55.632673 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt" event={"ID":"edf5288d-d3d4-4613-96cc-f12f166f4112","Type":"ContainerDied","Data":"b48f6c24b56442b4c26e5795d2c655d605e075491d914fa85fa84ac4718c1085"} Oct 30 22:52:55.632902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:52:55.632727 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b48f6c24b56442b4c26e5795d2c655d605e075491d914fa85fa84ac4718c1085" Oct 30 22:53:02.645533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.645501 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2"] Oct 30 22:53:02.645989 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.645974 4335 cpu_manager.go:414] "containerMap: removing container" podUID="edf5288d-d3d4-4613-96cc-f12f166f4112" containerName="revoke-keys" Oct 30 22:53:02.646037 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.645990 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf5288d-d3d4-4613-96cc-f12f166f4112" containerName="revoke-keys" Oct 30 22:53:02.646079 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.646062 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="edf5288d-d3d4-4613-96cc-f12f166f4112" containerName="revoke-keys" Oct 30 22:53:02.649315 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.649294 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2" Oct 30 22:53:02.661142 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.661119 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7645\" (UniqueName: \"kubernetes.io/projected/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-kube-api-access-k7645\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2\" (UID: \"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2" Oct 30 22:53:02.661243 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.661150 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2\" (UID: \"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2" Oct 30 22:53:02.661243 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.661184 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2\" (UID: \"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2" Oct 30 22:53:02.702482 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.702461 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:53:02.751830 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.751804 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:53:02.751830 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.751819 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:53:02.751990 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.751913 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:53:02.755134 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.755112 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2"] Oct 30 22:53:02.762886 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.762861 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7645\" (UniqueName: \"kubernetes.io/projected/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-kube-api-access-k7645\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2\" (UID: \"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2" Oct 30 22:53:02.762988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.762901 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2\" (UID: \"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2" Oct 30 22:53:02.762988 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.762949 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2\" (UID: \"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2" Oct 30 22:53:02.763524 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.763508 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2\" (UID: \"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2" Oct 30 22:53:02.765713 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.765684 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2\" (UID: \"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2" Oct 30 22:53:02.823385 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.823367 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7645\" (UniqueName: \"kubernetes.io/projected/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-kube-api-access-k7645\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2\" (UID: \"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2" Oct 30 22:53:02.962270 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:02.962216 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2" Oct 30 22:53:03.339739 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:03.339707 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2"] Oct 30 22:53:03.340721 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:53:03.340633 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf7a554_14b3_475e_9dc1_2ffbe72b4de2.slice/crio-032779c14ed4c865b6236fb7757a07cf502d0cedebddba1daa4ba4297854df40 WatchSource:0}: Error finding container 032779c14ed4c865b6236fb7757a07cf502d0cedebddba1daa4ba4297854df40: Status 404 returned error can't find the container with id 032779c14ed4c865b6236fb7757a07cf502d0cedebddba1daa4ba4297854df40 Oct 30 22:53:03.685036 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:03.685009 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2" event={"ID":"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2","Type":"ContainerStarted","Data":"df1b863bb74a9da9071484bd807feef7c98f2d684f044279de71b3e4ca43a09a"} Oct 30 22:53:03.685393 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:03.685054 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2" event={"ID":"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2","Type":"ContainerStarted","Data":"032779c14ed4c865b6236fb7757a07cf502d0cedebddba1daa4ba4297854df40"} Oct 30 22:53:03.801831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:03.801772 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2" podStartSLOduration=1.801757271 podStartE2EDuration="1.801757271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:03.800068398 +0000 UTC m=+4116.900500131" watchObservedRunningTime="2026-10-30 22:53:03.801757271 +0000 UTC m=+4116.902189021" Oct 30 22:53:04.046687 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:04.046599 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:53:04.692215 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:04.692169 4335 generic.go:356] "Generic (PLEG): container finished" podID="9bf7a554-14b3-475e-9dc1-2ffbe72b4de2" containerID="df1b863bb74a9da9071484bd807feef7c98f2d684f044279de71b3e4ca43a09a" exitCode=0 Oct 30 22:53:04.692646 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:04.692252 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2" event={"ID":"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2","Type":"ContainerDied","Data":"df1b863bb74a9da9071484bd807feef7c98f2d684f044279de71b3e4ca43a09a"} Oct 30 22:53:05.854448 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:05.854423 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2" Oct 30 22:53:05.891087 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:05.891065 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-kube-api-access-k7645\" (UniqueName: \"kubernetes.io/projected/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-kube-api-access-k7645\") pod \"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2\" (UID: \"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2\") " Oct 30 22:53:05.891199 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:05.891123 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-maas-api-openshift-ca\") pod \"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2\" (UID: \"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2\") " Oct 30 22:53:05.891199 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:05.891174 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-maas-api-xks-ca\") pod \"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2\" (UID: \"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2\") " Oct 30 22:53:05.891581 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:05.891550 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-maas-api-openshift-ca" pod "9bf7a554-14b3-475e-9dc1-2ffbe72b4de2" (UID: "9bf7a554-14b3-475e-9dc1-2ffbe72b4de2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:53:05.893344 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:05.893322 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-kube-api-access-k7645" pod "9bf7a554-14b3-475e-9dc1-2ffbe72b4de2" (UID: "9bf7a554-14b3-475e-9dc1-2ffbe72b4de2"). InnerVolumeSpecName "kube-api-access-k7645". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:05.893431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:05.893373 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-maas-api-xks-ca" pod "9bf7a554-14b3-475e-9dc1-2ffbe72b4de2" (UID: "9bf7a554-14b3-475e-9dc1-2ffbe72b4de2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:53:05.992416 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:05.992352 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:05.992416 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:05.992389 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7645\" (UniqueName: \"kubernetes.io/projected/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-kube-api-access-k7645\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:05.992416 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:05.992408 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:06.705050 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:06.705017 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2" event={"ID":"9bf7a554-14b3-475e-9dc1-2ffbe72b4de2","Type":"ContainerDied","Data":"032779c14ed4c865b6236fb7757a07cf502d0cedebddba1daa4ba4297854df40"} Oct 30 22:53:06.705229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:06.705053 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2" Oct 30 22:53:06.705229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:06.705063 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="032779c14ed4c865b6236fb7757a07cf502d0cedebddba1daa4ba4297854df40" Oct 30 22:53:07.103738 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:07.103709 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6"] Oct 30 22:53:11.061949 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:11.061920 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765"] Oct 30 22:53:11.110755 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:11.110729 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fac3c348-p7765"] Oct 30 22:53:13.056502 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:13.056468 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c502b64e-586e-4b56-9da1-73a4173c3084" path="/var/lib/kubelet/pods/c502b64e-586e-4b56-9da1-73a4173c3084/volumes" Oct 30 22:53:13.132112 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:13.132072 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d"] Oct 30 22:53:18.132984 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:18.132947 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" podUID="9ea43e1e-de80-4d46-bf75-e3e19bd17a5c" containerName="payload-pre-processing" containerID="cri-o://67bad79966aaee507861ebf0fc3c94d734b1fbc11d7c751207c33c71d28cd0f0" gracePeriod=25 Oct 30 22:53:22.124257 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.124215 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" podUID="5e07a315-b421-4dd2-8d3b-6d241c212377" containerName="main" containerID="cri-o://8ef46be4bdb499758d200ff6a921e75e47ff6e8360b518ee18c18d71a98cd710" gracePeriod=45 Oct 30 22:53:22.386311 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.386254 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:53:22.436390 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.436367 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e07a315-b421-4dd2-8d3b-6d241c212377-kube-api-access-j7m6v\" (UniqueName: \"kubernetes.io/projected/5e07a315-b421-4dd2-8d3b-6d241c212377-kube-api-access-j7m6v\") pod \"5e07a315-b421-4dd2-8d3b-6d241c212377\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " Oct 30 22:53:22.436515 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.436426 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e07a315-b421-4dd2-8d3b-6d241c212377-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e07a315-b421-4dd2-8d3b-6d241c212377-tls-certs\") pod \"5e07a315-b421-4dd2-8d3b-6d241c212377\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " Oct 30 22:53:22.436515 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.436510 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-home\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-home\") pod \"5e07a315-b421-4dd2-8d3b-6d241c212377\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " Oct 30 22:53:22.436605 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.436557 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-dshm\") pod \"5e07a315-b421-4dd2-8d3b-6d241c212377\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " Oct 30 22:53:22.436691 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.436646 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-tmp-dir\") pod \"5e07a315-b421-4dd2-8d3b-6d241c212377\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " Oct 30 22:53:22.436784 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.436767 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-model-cache\") pod \"5e07a315-b421-4dd2-8d3b-6d241c212377\" (UID: \"5e07a315-b421-4dd2-8d3b-6d241c212377\") " Oct 30 22:53:22.436915 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.436852 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-home" pod "5e07a315-b421-4dd2-8d3b-6d241c212377" (UID: "5e07a315-b421-4dd2-8d3b-6d241c212377"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:22.437075 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.436948 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-tmp-dir" pod "5e07a315-b421-4dd2-8d3b-6d241c212377" (UID: "5e07a315-b421-4dd2-8d3b-6d241c212377"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:22.437180 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.437057 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-model-cache" pod "5e07a315-b421-4dd2-8d3b-6d241c212377" (UID: "5e07a315-b421-4dd2-8d3b-6d241c212377"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:22.437281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.437212 4335 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-home\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:22.438841 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.438816 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e07a315-b421-4dd2-8d3b-6d241c212377-tls-certs" pod "5e07a315-b421-4dd2-8d3b-6d241c212377" (UID: "5e07a315-b421-4dd2-8d3b-6d241c212377"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:53:22.438938 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.438871 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-dshm" pod "5e07a315-b421-4dd2-8d3b-6d241c212377" (UID: "5e07a315-b421-4dd2-8d3b-6d241c212377"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:22.439012 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.438992 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e07a315-b421-4dd2-8d3b-6d241c212377-kube-api-access-j7m6v" pod "5e07a315-b421-4dd2-8d3b-6d241c212377" (UID: "5e07a315-b421-4dd2-8d3b-6d241c212377"). InnerVolumeSpecName "kube-api-access-j7m6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:22.537925 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.537888 4335 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-model-cache\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:22.537925 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.537923 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7m6v\" (UniqueName: \"kubernetes.io/projected/5e07a315-b421-4dd2-8d3b-6d241c212377-kube-api-access-j7m6v\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:22.538178 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.537940 4335 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e07a315-b421-4dd2-8d3b-6d241c212377-tls-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:22.538178 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.537954 4335 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-dshm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:22.538178 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.537968 4335 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e07a315-b421-4dd2-8d3b-6d241c212377-tmp-dir\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:22.804505 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.804471 4335 generic.go:356] "Generic (PLEG): container finished" podID="5e07a315-b421-4dd2-8d3b-6d241c212377" containerID="8ef46be4bdb499758d200ff6a921e75e47ff6e8360b518ee18c18d71a98cd710" exitCode=0 Oct 30 22:53:22.804748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.804512 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" Oct 30 22:53:22.804748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.804558 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" event={"ID":"5e07a315-b421-4dd2-8d3b-6d241c212377","Type":"ContainerDied","Data":"8ef46be4bdb499758d200ff6a921e75e47ff6e8360b518ee18c18d71a98cd710"} Oct 30 22:53:22.804748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.804599 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6" event={"ID":"5e07a315-b421-4dd2-8d3b-6d241c212377","Type":"ContainerDied","Data":"20877325e1c3f802018e271b3986eaec2a35dce44a8047677738ae16a5ce62e6"} Oct 30 22:53:22.804748 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.804651 4335 scope.go:122] "RemoveContainer" containerID="8ef46be4bdb499758d200ff6a921e75e47ff6e8360b518ee18c18d71a98cd710" Oct 30 22:53:22.818059 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.818035 4335 scope.go:122] "RemoveContainer" containerID="8ef46be4bdb499758d200ff6a921e75e47ff6e8360b518ee18c18d71a98cd710" Oct 30 22:53:22.818579 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:53:22.818545 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef46be4bdb499758d200ff6a921e75e47ff6e8360b518ee18c18d71a98cd710\": container with ID starting with 8ef46be4bdb499758d200ff6a921e75e47ff6e8360b518ee18c18d71a98cd710 not found: ID does not exist" containerID="8ef46be4bdb499758d200ff6a921e75e47ff6e8360b518ee18c18d71a98cd710" Oct 30 22:53:22.818786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.818750 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef46be4bdb499758d200ff6a921e75e47ff6e8360b518ee18c18d71a98cd710"} err="failed to get container status \"8ef46be4bdb499758d200ff6a921e75e47ff6e8360b518ee18c18d71a98cd710\": rpc error: code = NotFound desc = could not find container \"8ef46be4bdb499758d200ff6a921e75e47ff6e8360b518ee18c18d71a98cd710\": container with ID starting with 8ef46be4bdb499758d200ff6a921e75e47ff6e8360b518ee18c18d71a98cd710 not found: ID does not exist" Oct 30 22:53:22.969383 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:22.969359 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6"] Oct 30 22:53:23.019375 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:23.019350 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-bf99ca/simulator-single-main-bf99ca-kserve-866c9c7b76-sdhp6"] Oct 30 22:53:23.057362 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:23.057308 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e07a315-b421-4dd2-8d3b-6d241c212377" path="/var/lib/kubelet/pods/5e07a315-b421-4dd2-8d3b-6d241c212377/volumes" Oct 30 22:53:23.188087 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:23.188062 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" podUID="9ea43e1e-de80-4d46-bf75-e3e19bd17a5c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:53:23.533384 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:23.533304 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2"] Oct 30 22:53:23.536096 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:23.536071 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bf99ca-27wb2"] Oct 30 22:53:25.056576 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:25.056540 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf7a554-14b3-475e-9dc1-2ffbe72b4de2" path="/var/lib/kubelet/pods/9bf7a554-14b3-475e-9dc1-2ffbe72b4de2/volumes" Oct 30 22:53:25.321281 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:53:25.321208 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:53:26.335466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:26.335436 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:53:33.188498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:33.188457 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" podUID="9ea43e1e-de80-4d46-bf75-e3e19bd17a5c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:53:37.154103 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.154058 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj"] Oct 30 22:53:37.154495 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.154482 4335 cpu_manager.go:414] "containerMap: removing container" podUID="5e07a315-b421-4dd2-8d3b-6d241c212377" containerName="main" Oct 30 22:53:37.154543 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.154495 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e07a315-b421-4dd2-8d3b-6d241c212377" containerName="main" Oct 30 22:53:37.154543 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.154519 4335 cpu_manager.go:414] "containerMap: removing container" podUID="9bf7a554-14b3-475e-9dc1-2ffbe72b4de2" containerName="revoke-keys" Oct 30 22:53:37.154543 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.154525 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf7a554-14b3-475e-9dc1-2ffbe72b4de2" containerName="revoke-keys" Oct 30 22:53:37.154679 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.154589 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e07a315-b421-4dd2-8d3b-6d241c212377" containerName="main" Oct 30 22:53:37.154679 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.154601 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf7a554-14b3-475e-9dc1-2ffbe72b4de2" containerName="revoke-keys" Oct 30 22:53:37.157792 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.157775 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.215108 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.215075 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj"] Oct 30 22:53:37.260814 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.260790 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-f5d67e12-kserve-self-signed-certs\"" Oct 30 22:53:37.263054 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.263033 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-tmp-dir\") pod \"e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.263158 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.263066 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2vlp\" (UniqueName: \"kubernetes.io/projected/902234b4-43b0-4139-9dc3-87543c0c79cf-kube-api-access-p2vlp\") pod \"e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.263158 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.263097 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902234b4-43b0-4139-9dc3-87543c0c79cf-tls-certs\") pod \"e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.263243 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.263179 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-model-cache\") pod \"e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.263279 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.263258 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-home\") pod \"e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.263314 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.263283 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-dshm\") pod \"e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.364378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.364344 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-tmp-dir\") pod \"e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.364378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.364384 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2vlp\" (UniqueName: \"kubernetes.io/projected/902234b4-43b0-4139-9dc3-87543c0c79cf-kube-api-access-p2vlp\") pod \"e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.364686 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.364429 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902234b4-43b0-4139-9dc3-87543c0c79cf-tls-certs\") pod \"e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.364686 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.364461 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-model-cache\") pod \"e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.364686 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.364520 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-home\") pod \"e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.364686 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.364543 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-dshm\") pod \"e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.364939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.364841 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-tmp-dir\") pod \"e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.364939 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.364915 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-model-cache\") pod \"e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.365033 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.365012 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-home\") pod \"e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.367006 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.366985 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-dshm\") pod \"e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.367129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.367112 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902234b4-43b0-4139-9dc3-87543c0c79cf-tls-certs\") pod \"e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.425263 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.425210 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2vlp\" (UniqueName: \"kubernetes.io/projected/902234b4-43b0-4139-9dc3-87543c0c79cf-kube-api-access-p2vlp\") pod \"e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.468842 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.468816 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:37.853631 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.853590 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj"] Oct 30 22:53:37.854576 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:53:37.854544 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod902234b4_43b0_4139_9dc3_87543c0c79cf.slice/crio-394aac7700d85f62b85aa19600d1b2044786a805f895857325713abaa7478e33 WatchSource:0}: Error finding container 394aac7700d85f62b85aa19600d1b2044786a805f895857325713abaa7478e33: Status 404 returned error can't find the container with id 394aac7700d85f62b85aa19600d1b2044786a805f895857325713abaa7478e33 Oct 30 22:53:37.901131 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:37.901106 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" event={"ID":"902234b4-43b0-4139-9dc3-87543c0c79cf","Type":"ContainerStarted","Data":"394aac7700d85f62b85aa19600d1b2044786a805f895857325713abaa7478e33"} Oct 30 22:53:38.356005 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:38.355982 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" Oct 30 22:53:38.475467 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:38.475113 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ea43e1e-de80-4d46-bf75-e3e19bd17a5c-kube-api-access-mcttq\" (UniqueName: \"kubernetes.io/projected/9ea43e1e-de80-4d46-bf75-e3e19bd17a5c-kube-api-access-mcttq\") pod \"9ea43e1e-de80-4d46-bf75-e3e19bd17a5c\" (UID: \"9ea43e1e-de80-4d46-bf75-e3e19bd17a5c\") " Oct 30 22:53:38.475467 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:38.475195 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ea43e1e-de80-4d46-bf75-e3e19bd17a5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea43e1e-de80-4d46-bf75-e3e19bd17a5c-plugins-config-volume\") pod \"9ea43e1e-de80-4d46-bf75-e3e19bd17a5c\" (UID: \"9ea43e1e-de80-4d46-bf75-e3e19bd17a5c\") " Oct 30 22:53:38.475847 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:38.475830 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea43e1e-de80-4d46-bf75-e3e19bd17a5c-plugins-config-volume" pod "9ea43e1e-de80-4d46-bf75-e3e19bd17a5c" (UID: "9ea43e1e-de80-4d46-bf75-e3e19bd17a5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:53:38.478389 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:38.478364 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea43e1e-de80-4d46-bf75-e3e19bd17a5c-kube-api-access-mcttq" pod "9ea43e1e-de80-4d46-bf75-e3e19bd17a5c" (UID: "9ea43e1e-de80-4d46-bf75-e3e19bd17a5c"). InnerVolumeSpecName "kube-api-access-mcttq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:38.576512 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:38.576479 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcttq\" (UniqueName: \"kubernetes.io/projected/9ea43e1e-de80-4d46-bf75-e3e19bd17a5c-kube-api-access-mcttq\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:38.576512 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:38.576505 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea43e1e-de80-4d46-bf75-e3e19bd17a5c-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:53:38.908349 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:38.908315 4335 generic.go:356] "Generic (PLEG): container finished" podID="9ea43e1e-de80-4d46-bf75-e3e19bd17a5c" containerID="67bad79966aaee507861ebf0fc3c94d734b1fbc11d7c751207c33c71d28cd0f0" exitCode=0 Oct 30 22:53:38.908534 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:38.908383 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" Oct 30 22:53:38.908534 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:38.908409 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" event={"ID":"9ea43e1e-de80-4d46-bf75-e3e19bd17a5c","Type":"ContainerDied","Data":"67bad79966aaee507861ebf0fc3c94d734b1fbc11d7c751207c33c71d28cd0f0"} Oct 30 22:53:38.908534 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:38.908486 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d" event={"ID":"9ea43e1e-de80-4d46-bf75-e3e19bd17a5c","Type":"ContainerDied","Data":"004d03f76fac0009b70adbab83839403e77f89f54a92676afaf164c4d12e1604"} Oct 30 22:53:38.908534 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:38.908518 4335 scope.go:122] "RemoveContainer" containerID="67bad79966aaee507861ebf0fc3c94d734b1fbc11d7c751207c33c71d28cd0f0" Oct 30 22:53:38.910168 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:38.910146 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" event={"ID":"902234b4-43b0-4139-9dc3-87543c0c79cf","Type":"ContainerStarted","Data":"b97961bd6c45b94713514601d70f855c5512d87202d17130f86e406b79325988"} Oct 30 22:53:38.918404 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:38.918384 4335 scope.go:122] "RemoveContainer" containerID="67bad79966aaee507861ebf0fc3c94d734b1fbc11d7c751207c33c71d28cd0f0" Oct 30 22:53:38.918676 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:53:38.918657 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67bad79966aaee507861ebf0fc3c94d734b1fbc11d7c751207c33c71d28cd0f0\": container with ID starting with 67bad79966aaee507861ebf0fc3c94d734b1fbc11d7c751207c33c71d28cd0f0 not found: ID does not exist" containerID="67bad79966aaee507861ebf0fc3c94d734b1fbc11d7c751207c33c71d28cd0f0" Oct 30 22:53:38.918740 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:38.918686 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67bad79966aaee507861ebf0fc3c94d734b1fbc11d7c751207c33c71d28cd0f0"} err="failed to get container status \"67bad79966aaee507861ebf0fc3c94d734b1fbc11d7c751207c33c71d28cd0f0\": rpc error: code = NotFound desc = could not find container \"67bad79966aaee507861ebf0fc3c94d734b1fbc11d7c751207c33c71d28cd0f0\": container with ID starting with 67bad79966aaee507861ebf0fc3c94d734b1fbc11d7c751207c33c71d28cd0f0 not found: ID does not exist" Oct 30 22:53:39.076541 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:39.076501 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" podStartSLOduration=2.076489763 podStartE2EDuration="2.076489763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:39.076009523 +0000 UTC m=+4152.176441254" watchObservedRunningTime="2026-10-30 22:53:39.076489763 +0000 UTC m=+4152.176921495" Oct 30 22:53:39.193450 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:39.189324 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d"] Oct 30 22:53:39.198217 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:39.198186 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf99ca-8786864r6d"] Oct 30 22:53:39.469556 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:39.469492 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:41.056301 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:41.056268 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ea43e1e-de80-4d46-bf75-e3e19bd17a5c" path="/var/lib/kubelet/pods/9ea43e1e-de80-4d46-bf75-e3e19bd17a5c/volumes" Oct 30 22:53:41.854888 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:41.854856 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:53:47.469780 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:47.469740 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:47.482222 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:47.482193 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:48.018163 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:48.018114 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:53:49.048609 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:49.048581 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht"] Oct 30 22:53:49.097593 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:49.097565 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vrwht"] Oct 30 22:53:51.055829 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:53:51.055785 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7481e763-3843-43d2-9731-f5d6c13f62b0" path="/var/lib/kubelet/pods/7481e763-3843-43d2-9731-f5d6c13f62b0/volumes" Oct 30 22:53:55.320498 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:53:55.320462 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:54:20.845068 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.844987 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572"] Oct 30 22:54:20.845465 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.845442 4335 cpu_manager.go:414] "containerMap: removing container" podUID="9ea43e1e-de80-4d46-bf75-e3e19bd17a5c" containerName="payload-pre-processing" Oct 30 22:54:20.845465 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.845457 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ea43e1e-de80-4d46-bf75-e3e19bd17a5c" containerName="payload-pre-processing" Oct 30 22:54:20.845569 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.845556 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ea43e1e-de80-4d46-bf75-e3e19bd17a5c" containerName="payload-pre-processing" Oct 30 22:54:20.848985 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.848968 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:20.873213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.873189 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-dshm\") pod \"e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:20.873328 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.873227 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-tmp-dir\") pod \"e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:20.873328 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.873253 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbhd9\" (UniqueName: \"kubernetes.io/projected/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-kube-api-access-pbhd9\") pod \"e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:20.873405 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.873389 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-model-cache\") pod \"e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:20.873444 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.873418 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-tls-certs\") pod \"e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:20.873479 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.873456 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-home\") pod \"e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:20.952006 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.951972 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-f5d67e12-kserve-self-signed-certs\"" Oct 30 22:54:20.955906 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.955877 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572"] Oct 30 22:54:20.974259 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.974235 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-dshm\") pod \"e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:20.974410 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.974273 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-tmp-dir\") pod \"e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:20.974410 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.974305 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbhd9\" (UniqueName: \"kubernetes.io/projected/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-kube-api-access-pbhd9\") pod \"e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:20.974410 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.974353 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-model-cache\") pod \"e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:20.974410 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.974382 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-tls-certs\") pod \"e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:20.974697 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.974425 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-home\") pod \"e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:20.974817 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.974792 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-tmp-dir\") pod \"e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:20.974893 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.974834 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-model-cache\") pod \"e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:20.974893 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.974869 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-home\") pod \"e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:20.976710 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.976681 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-dshm\") pod \"e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:20.976970 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:20.976953 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-tls-certs\") pod \"e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:21.035054 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:21.035024 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbhd9\" (UniqueName: \"kubernetes.io/projected/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-kube-api-access-pbhd9\") pod \"e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:21.161435 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:21.161368 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:21.549444 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:21.549414 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572"] Oct 30 22:54:21.551197 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:54:21.551160 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b605f0f_5de4_44c6_9a27_ededc9bbd5cd.slice/crio-2d71219102cd23a28ba49b7e6a20e9a1f163239b04acd7808aad299bb9dde13e WatchSource:0}: Error finding container 2d71219102cd23a28ba49b7e6a20e9a1f163239b04acd7808aad299bb9dde13e: Status 404 returned error can't find the container with id 2d71219102cd23a28ba49b7e6a20e9a1f163239b04acd7808aad299bb9dde13e Oct 30 22:54:22.219713 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:22.219670 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" event={"ID":"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd","Type":"ContainerStarted","Data":"0a21f1a9da6b7721f0c86bde0d2175bfe2fc60ae07e96b81230e0c7e46f1eacd"} Oct 30 22:54:22.220108 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:22.219724 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" event={"ID":"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd","Type":"ContainerStarted","Data":"2d71219102cd23a28ba49b7e6a20e9a1f163239b04acd7808aad299bb9dde13e"} Oct 30 22:54:22.337320 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:22.337109 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" podStartSLOduration=2.3370914689999998 podStartE2EDuration="2.337091469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:54:22.336578712 +0000 UTC m=+4195.437010443" watchObservedRunningTime="2026-10-30 22:54:22.337091469 +0000 UTC m=+4195.437523203" Oct 30 22:54:23.162325 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:23.162296 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:25.322341 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:54:25.322300 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:54:25.741103 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:25.741030 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:54:28.726640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:28.726046 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:54:28.935135 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:28.935088 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bbbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:54:28.935449 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:28.935415 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" podUID="4e6addff-b99f-4e05-a17d-00bb0f88017c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:54:28.937325 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:28.937244 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:54:31.162359 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:31.162326 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:31.171313 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:31.171285 4335 scope.go:122] "RemoveContainer" containerID="5546746cbef39850fdac379616c73f3a68498a069563a8fc3069715a2ca2299d" Oct 30 22:54:31.175367 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:31.175347 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:31.181116 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:31.181096 4335 scope.go:122] "RemoveContainer" containerID="adc49b0997a3ccc41a271dbed589d810f096e7b6232d7a6d10e8250aa2d9596b" Oct 30 22:54:31.357505 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:31.357476 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:54:54.518307 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:54.518273 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j"] Oct 30 22:54:54.522088 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:54.522066 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j" Oct 30 22:54:54.576870 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:54.576826 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:54:54.581294 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:54.581265 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j"] Oct 30 22:54:54.622363 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:54.622334 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960b0f4e-0d48-4009-b141-19579f9963d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-x7z9j\" (UID: \"960b0f4e-0d48-4009-b141-19579f9963d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j" Oct 30 22:54:54.622505 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:54.622383 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk246\" (UniqueName: \"kubernetes.io/projected/960b0f4e-0d48-4009-b141-19579f9963d0-kube-api-access-dk246\") pod \"maas-api-revoke-sub-452b6d7b-x7z9j\" (UID: \"960b0f4e-0d48-4009-b141-19579f9963d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j" Oct 30 22:54:54.622505 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:54.622459 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960b0f4e-0d48-4009-b141-19579f9963d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-x7z9j\" (UID: \"960b0f4e-0d48-4009-b141-19579f9963d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j" Oct 30 22:54:54.626247 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:54.626227 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:54:54.626322 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:54.626246 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:54:54.626322 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:54.626248 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:54:54.723721 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:54.723698 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960b0f4e-0d48-4009-b141-19579f9963d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-x7z9j\" (UID: \"960b0f4e-0d48-4009-b141-19579f9963d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j" Oct 30 22:54:54.723839 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:54.723782 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960b0f4e-0d48-4009-b141-19579f9963d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-x7z9j\" (UID: \"960b0f4e-0d48-4009-b141-19579f9963d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j" Oct 30 22:54:54.723839 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:54.723824 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk246\" (UniqueName: \"kubernetes.io/projected/960b0f4e-0d48-4009-b141-19579f9963d0-kube-api-access-dk246\") pod \"maas-api-revoke-sub-452b6d7b-x7z9j\" (UID: \"960b0f4e-0d48-4009-b141-19579f9963d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j" Oct 30 22:54:54.724398 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:54.724376 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960b0f4e-0d48-4009-b141-19579f9963d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-x7z9j\" (UID: \"960b0f4e-0d48-4009-b141-19579f9963d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j" Oct 30 22:54:54.726195 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:54.726173 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960b0f4e-0d48-4009-b141-19579f9963d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-x7z9j\" (UID: \"960b0f4e-0d48-4009-b141-19579f9963d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j" Oct 30 22:54:54.784450 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:54.784397 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk246\" (UniqueName: \"kubernetes.io/projected/960b0f4e-0d48-4009-b141-19579f9963d0-kube-api-access-dk246\") pod \"maas-api-revoke-sub-452b6d7b-x7z9j\" (UID: \"960b0f4e-0d48-4009-b141-19579f9963d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j" Oct 30 22:54:54.836538 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:54.836519 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j" Oct 30 22:54:55.209214 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:55.209187 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j"] Oct 30 22:54:55.209862 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:54:55.209823 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960b0f4e_0d48_4009_b141_19579f9963d0.slice/crio-f9fa939d8415910f5253f7c56cbe5bf0a16bbc7bca5f1ab097129ec487c43a3d WatchSource:0}: Error finding container f9fa939d8415910f5253f7c56cbe5bf0a16bbc7bca5f1ab097129ec487c43a3d: Status 404 returned error can't find the container with id f9fa939d8415910f5253f7c56cbe5bf0a16bbc7bca5f1ab097129ec487c43a3d Oct 30 22:54:55.322389 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:54:55.322354 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:54:55.489285 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:55.489221 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j" event={"ID":"960b0f4e-0d48-4009-b141-19579f9963d0","Type":"ContainerStarted","Data":"f9fa939d8415910f5253f7c56cbe5bf0a16bbc7bca5f1ab097129ec487c43a3d"} Oct 30 22:54:56.041198 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:56.041169 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:54:56.496431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:56.496402 4335 generic.go:356] "Generic (PLEG): container finished" podID="960b0f4e-0d48-4009-b141-19579f9963d0" containerID="e2bab0a64c7e168aacdc0879583087e8286dcab8285edd51df3524dd9cf4f404" exitCode=0 Oct 30 22:54:56.496583 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:56.496475 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j" event={"ID":"960b0f4e-0d48-4009-b141-19579f9963d0","Type":"ContainerDied","Data":"e2bab0a64c7e168aacdc0879583087e8286dcab8285edd51df3524dd9cf4f404"} Oct 30 22:54:57.907239 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:57.906950 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j" Oct 30 22:54:58.109700 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:58.092943 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960b0f4e-0d48-4009-b141-19579f9963d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960b0f4e-0d48-4009-b141-19579f9963d0-maas-api-openshift-ca\") pod \"960b0f4e-0d48-4009-b141-19579f9963d0\" (UID: \"960b0f4e-0d48-4009-b141-19579f9963d0\") " Oct 30 22:54:58.130673 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:58.109969 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/960b0f4e-0d48-4009-b141-19579f9963d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960b0f4e-0d48-4009-b141-19579f9963d0-maas-api-xks-ca\") pod \"960b0f4e-0d48-4009-b141-19579f9963d0\" (UID: \"960b0f4e-0d48-4009-b141-19579f9963d0\") " Oct 30 22:54:58.239846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:58.239815 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960b0f4e-0d48-4009-b141-19579f9963d0-kube-api-access-dk246\" (UniqueName: \"kubernetes.io/projected/960b0f4e-0d48-4009-b141-19579f9963d0-kube-api-access-dk246\") pod \"960b0f4e-0d48-4009-b141-19579f9963d0\" (UID: \"960b0f4e-0d48-4009-b141-19579f9963d0\") " Oct 30 22:54:58.258784 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:58.116821 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960b0f4e-0d48-4009-b141-19579f9963d0-maas-api-openshift-ca" pod "960b0f4e-0d48-4009-b141-19579f9963d0" (UID: "960b0f4e-0d48-4009-b141-19579f9963d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:54:58.258989 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:58.192181 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/960b0f4e-0d48-4009-b141-19579f9963d0-maas-api-xks-ca" pod "960b0f4e-0d48-4009-b141-19579f9963d0" (UID: "960b0f4e-0d48-4009-b141-19579f9963d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:54:58.326828 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:58.326264 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960b0f4e-0d48-4009-b141-19579f9963d0-kube-api-access-dk246" pod "960b0f4e-0d48-4009-b141-19579f9963d0" (UID: "960b0f4e-0d48-4009-b141-19579f9963d0"). InnerVolumeSpecName "kube-api-access-dk246". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:58.359638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:58.359265 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk246\" (UniqueName: \"kubernetes.io/projected/960b0f4e-0d48-4009-b141-19579f9963d0-kube-api-access-dk246\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:58.359638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:58.359299 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960b0f4e-0d48-4009-b141-19579f9963d0-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:58.359638 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:58.359315 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960b0f4e-0d48-4009-b141-19579f9963d0-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:54:58.528913 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:58.528824 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j" event={"ID":"960b0f4e-0d48-4009-b141-19579f9963d0","Type":"ContainerDied","Data":"f9fa939d8415910f5253f7c56cbe5bf0a16bbc7bca5f1ab097129ec487c43a3d"} Oct 30 22:54:58.529153 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:58.529133 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9fa939d8415910f5253f7c56cbe5bf0a16bbc7bca5f1ab097129ec487c43a3d" Oct 30 22:54:58.529354 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:58.529338 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j" Oct 30 22:54:58.935754 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:58.935702 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bbbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:54:58.936403 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:54:58.936351 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" podUID="4e6addff-b99f-4e05-a17d-00bb0f88017c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:55:07.352315 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:07.352284 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj"] Oct 30 22:55:11.365419 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:11.365389 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:55:12.076768 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:12.076729 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572"] Oct 30 22:55:13.935843 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:13.935701 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bbbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:55:13.936895 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:13.936838 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bbbj" podUID="4e6addff-b99f-4e05-a17d-00bb0f88017c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:55:16.191592 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.191558 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv"] Oct 30 22:55:16.192034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.192018 4335 cpu_manager.go:414] "containerMap: removing container" podUID="960b0f4e-0d48-4009-b141-19579f9963d0" containerName="revoke-keys" Oct 30 22:55:16.192084 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.192036 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960b0f4e-0d48-4009-b141-19579f9963d0" containerName="revoke-keys" Oct 30 22:55:16.192130 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.192121 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="960b0f4e-0d48-4009-b141-19579f9963d0" containerName="revoke-keys" Oct 30 22:55:16.195309 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.195295 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv" Oct 30 22:55:16.244496 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.244470 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-zc7bv\" (UID: \"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv" Oct 30 22:55:16.244652 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.244511 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-zc7bv\" (UID: \"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv" Oct 30 22:55:16.244729 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.244708 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmhgg\" (UniqueName: \"kubernetes.io/projected/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-kube-api-access-dmhgg\") pod \"maas-api-revoke-sub-3e31a506-zc7bv\" (UID: \"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv" Oct 30 22:55:16.299684 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.299645 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:55:16.299835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.299700 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:55:16.299835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.299700 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:55:16.299835 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.299700 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:55:16.302703 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.302683 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv"] Oct 30 22:55:16.345751 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.345730 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmhgg\" (UniqueName: \"kubernetes.io/projected/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-kube-api-access-dmhgg\") pod \"maas-api-revoke-sub-3e31a506-zc7bv\" (UID: \"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv" Oct 30 22:55:16.345877 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.345766 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-zc7bv\" (UID: \"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv" Oct 30 22:55:16.345877 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.345796 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-zc7bv\" (UID: \"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv" Oct 30 22:55:16.346398 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.346378 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-zc7bv\" (UID: \"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv" Oct 30 22:55:16.348246 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.348227 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-zc7bv\" (UID: \"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv" Oct 30 22:55:16.406970 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.406947 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmhgg\" (UniqueName: \"kubernetes.io/projected/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-kube-api-access-dmhgg\") pod \"maas-api-revoke-sub-3e31a506-zc7bv\" (UID: \"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv" Oct 30 22:55:16.516074 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.516025 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv" Oct 30 22:55:16.891827 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:55:16.891794 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a49a8c2_6933_4e2f_a2aa_2abb5a7db9af.slice/crio-7c8af6e04fbbe3463b0ae8cdf477d965020a50843f0b03bdd4d4ecdfcda7a5d1 WatchSource:0}: Error finding container 7c8af6e04fbbe3463b0ae8cdf477d965020a50843f0b03bdd4d4ecdfcda7a5d1: Status 404 returned error can't find the container with id 7c8af6e04fbbe3463b0ae8cdf477d965020a50843f0b03bdd4d4ecdfcda7a5d1 Oct 30 22:55:16.892017 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:16.891978 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv"] Oct 30 22:55:17.701834 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:17.701800 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv" event={"ID":"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af","Type":"ContainerStarted","Data":"f656ae526ea73cf628d38ed0583b86c205a634c08eb55a92e32d09a61389b438"} Oct 30 22:55:17.702194 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:17.701877 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv" event={"ID":"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af","Type":"ContainerStarted","Data":"7c8af6e04fbbe3463b0ae8cdf477d965020a50843f0b03bdd4d4ecdfcda7a5d1"} Oct 30 22:55:17.819505 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:17.819452 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv" podStartSLOduration=1.819437716 podStartE2EDuration="1.819437716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:17.816690635 +0000 UTC m=+4250.917122358" watchObservedRunningTime="2026-10-30 22:55:17.819437716 +0000 UTC m=+4250.919869447" Oct 30 22:55:18.709170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:18.709136 4335 generic.go:356] "Generic (PLEG): container finished" podID="3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af" containerID="f656ae526ea73cf628d38ed0583b86c205a634c08eb55a92e32d09a61389b438" exitCode=0 Oct 30 22:55:18.709170 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:18.709173 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv" event={"ID":"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af","Type":"ContainerDied","Data":"f656ae526ea73cf628d38ed0583b86c205a634c08eb55a92e32d09a61389b438"} Oct 30 22:55:19.863130 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:19.863107 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv" Oct 30 22:55:19.985429 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:19.985361 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-maas-api-xks-ca\") pod \"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af\" (UID: \"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af\") " Oct 30 22:55:19.985583 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:19.985464 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-maas-api-openshift-ca\") pod \"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af\" (UID: \"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af\") " Oct 30 22:55:19.985583 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:19.985489 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-kube-api-access-dmhgg\" (UniqueName: \"kubernetes.io/projected/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-kube-api-access-dmhgg\") pod \"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af\" (UID: \"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af\") " Oct 30 22:55:19.985902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:19.985873 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-maas-api-openshift-ca" pod "3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af" (UID: "3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:55:19.987665 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:19.987627 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-maas-api-xks-ca" pod "3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af" (UID: "3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:55:19.987801 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:19.987773 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-kube-api-access-dmhgg" pod "3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af" (UID: "3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af"). InnerVolumeSpecName "kube-api-access-dmhgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:20.086409 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:20.086377 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:20.086409 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:20.086404 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmhgg\" (UniqueName: \"kubernetes.io/projected/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-kube-api-access-dmhgg\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:20.086409 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:20.086414 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:20.726129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:20.726093 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv" Oct 30 22:55:20.726342 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:20.726159 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv" event={"ID":"3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af","Type":"ContainerDied","Data":"7c8af6e04fbbe3463b0ae8cdf477d965020a50843f0b03bdd4d4ecdfcda7a5d1"} Oct 30 22:55:20.726342 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:20.726215 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c8af6e04fbbe3463b0ae8cdf477d965020a50843f0b03bdd4d4ecdfcda7a5d1" Oct 30 22:55:22.438937 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:22.438768 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" podUID="902234b4-43b0-4139-9dc3-87543c0c79cf" containerName="main" containerID="cri-o://b97961bd6c45b94713514601d70f855c5512d87202d17130f86e406b79325988" gracePeriod=45 Oct 30 22:55:23.011698 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.008011 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" podUID="902234b4-43b0-4139-9dc3-87543c0c79cf" containerName="main" probeResult="failure" output="Get \"https://10.133.1.50:8000/ready\": dial tcp 10.133.1.50:8000: connect: connection refused" Oct 30 22:55:23.385235 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.384919 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:55:23.498337 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.498311 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-home\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-home\") pod \"902234b4-43b0-4139-9dc3-87543c0c79cf\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " Oct 30 22:55:23.498863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.498355 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-dshm\") pod \"902234b4-43b0-4139-9dc3-87543c0c79cf\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " Oct 30 22:55:23.498863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.498412 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-model-cache\") pod \"902234b4-43b0-4139-9dc3-87543c0c79cf\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " Oct 30 22:55:23.498863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.498474 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902234b4-43b0-4139-9dc3-87543c0c79cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/902234b4-43b0-4139-9dc3-87543c0c79cf-tls-certs\") pod \"902234b4-43b0-4139-9dc3-87543c0c79cf\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " Oct 30 22:55:23.498863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.498522 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902234b4-43b0-4139-9dc3-87543c0c79cf-kube-api-access-p2vlp\" (UniqueName: \"kubernetes.io/projected/902234b4-43b0-4139-9dc3-87543c0c79cf-kube-api-access-p2vlp\") pod \"902234b4-43b0-4139-9dc3-87543c0c79cf\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " Oct 30 22:55:23.498863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.498547 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-tmp-dir\") pod \"902234b4-43b0-4139-9dc3-87543c0c79cf\" (UID: \"902234b4-43b0-4139-9dc3-87543c0c79cf\") " Oct 30 22:55:23.498863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.498641 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-home" pod "902234b4-43b0-4139-9dc3-87543c0c79cf" (UID: "902234b4-43b0-4139-9dc3-87543c0c79cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:23.498863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.498658 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-model-cache" pod "902234b4-43b0-4139-9dc3-87543c0c79cf" (UID: "902234b4-43b0-4139-9dc3-87543c0c79cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:23.498863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.498799 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-tmp-dir" pod "902234b4-43b0-4139-9dc3-87543c0c79cf" (UID: "902234b4-43b0-4139-9dc3-87543c0c79cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:23.499260 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.498925 4335 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-tmp-dir\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:23.499260 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.498944 4335 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-home\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:23.499260 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.498954 4335 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-model-cache\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:23.501058 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.501033 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-dshm" pod "902234b4-43b0-4139-9dc3-87543c0c79cf" (UID: "902234b4-43b0-4139-9dc3-87543c0c79cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:23.501431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.501405 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902234b4-43b0-4139-9dc3-87543c0c79cf-tls-certs" pod "902234b4-43b0-4139-9dc3-87543c0c79cf" (UID: "902234b4-43b0-4139-9dc3-87543c0c79cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:55:23.501719 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.501694 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902234b4-43b0-4139-9dc3-87543c0c79cf-kube-api-access-p2vlp" pod "902234b4-43b0-4139-9dc3-87543c0c79cf" (UID: "902234b4-43b0-4139-9dc3-87543c0c79cf"). InnerVolumeSpecName "kube-api-access-p2vlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:23.599934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.599880 4335 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902234b4-43b0-4139-9dc3-87543c0c79cf-dshm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:23.599934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.599901 4335 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902234b4-43b0-4139-9dc3-87543c0c79cf-tls-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:23.599934 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.599910 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2vlp\" (UniqueName: \"kubernetes.io/projected/902234b4-43b0-4139-9dc3-87543c0c79cf-kube-api-access-p2vlp\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:23.815986 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.815960 4335 generic.go:356] "Generic (PLEG): container finished" podID="902234b4-43b0-4139-9dc3-87543c0c79cf" containerID="b97961bd6c45b94713514601d70f855c5512d87202d17130f86e406b79325988" exitCode=0 Oct 30 22:55:23.816113 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.815999 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" Oct 30 22:55:23.816113 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.816059 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" event={"ID":"902234b4-43b0-4139-9dc3-87543c0c79cf","Type":"ContainerDied","Data":"b97961bd6c45b94713514601d70f855c5512d87202d17130f86e406b79325988"} Oct 30 22:55:23.816113 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.816105 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj" event={"ID":"902234b4-43b0-4139-9dc3-87543c0c79cf","Type":"ContainerDied","Data":"394aac7700d85f62b85aa19600d1b2044786a805f895857325713abaa7478e33"} Oct 30 22:55:23.816224 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.816124 4335 scope.go:122] "RemoveContainer" containerID="b97961bd6c45b94713514601d70f855c5512d87202d17130f86e406b79325988" Oct 30 22:55:23.827569 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.827553 4335 scope.go:122] "RemoveContainer" containerID="b97961bd6c45b94713514601d70f855c5512d87202d17130f86e406b79325988" Oct 30 22:55:23.827883 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:55:23.827864 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b97961bd6c45b94713514601d70f855c5512d87202d17130f86e406b79325988\": container with ID starting with b97961bd6c45b94713514601d70f855c5512d87202d17130f86e406b79325988 not found: ID does not exist" containerID="b97961bd6c45b94713514601d70f855c5512d87202d17130f86e406b79325988" Oct 30 22:55:23.827957 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.827891 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b97961bd6c45b94713514601d70f855c5512d87202d17130f86e406b79325988"} err="failed to get container status \"b97961bd6c45b94713514601d70f855c5512d87202d17130f86e406b79325988\": rpc error: code = NotFound desc = could not find container \"b97961bd6c45b94713514601d70f855c5512d87202d17130f86e406b79325988\": container with ID starting with b97961bd6c45b94713514601d70f855c5512d87202d17130f86e406b79325988 not found: ID does not exist" Oct 30 22:55:23.983959 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:23.983907 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj"] Oct 30 22:55:24.035036 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:24.035000 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-f5d67e12-kserve-9fd958857-k9spj"] Oct 30 22:55:25.057928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:25.057889 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="902234b4-43b0-4139-9dc3-87543c0c79cf" path="/var/lib/kubelet/pods/902234b4-43b0-4139-9dc3-87543c0c79cf/volumes" Oct 30 22:55:25.319644 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:55:25.319552 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:55:27.097593 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.097547 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" podUID="0b605f0f-5de4-44c6-9a27-ededc9bbd5cd" containerName="main" containerID="cri-o://0a21f1a9da6b7721f0c86bde0d2175bfe2fc60ae07e96b81230e0c7e46f1eacd" gracePeriod=45 Oct 30 22:55:27.359655 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.359582 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:55:27.532800 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.532762 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-home\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-home\") pod \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " Oct 30 22:55:27.532969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.532824 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-tmp-dir\") pod \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " Oct 30 22:55:27.532969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.532850 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-kube-api-access-pbhd9\" (UniqueName: \"kubernetes.io/projected/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-kube-api-access-pbhd9\") pod \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " Oct 30 22:55:27.532969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.532894 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-dshm\") pod \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " Oct 30 22:55:27.532969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.532959 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-tls-certs\") pod \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " Oct 30 22:55:27.533210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.533004 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-home" pod "0b605f0f-5de4-44c6-9a27-ededc9bbd5cd" (UID: "0b605f0f-5de4-44c6-9a27-ededc9bbd5cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:27.533210 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.533079 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-model-cache\") pod \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\" (UID: \"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd\") " Oct 30 22:55:27.533400 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.533372 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-model-cache" pod "0b605f0f-5de4-44c6-9a27-ededc9bbd5cd" (UID: "0b605f0f-5de4-44c6-9a27-ededc9bbd5cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:27.533494 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.533407 4335 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-home\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:27.533494 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.533410 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-tmp-dir" pod "0b605f0f-5de4-44c6-9a27-ededc9bbd5cd" (UID: "0b605f0f-5de4-44c6-9a27-ededc9bbd5cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:27.535213 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.535188 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-dshm" pod "0b605f0f-5de4-44c6-9a27-ededc9bbd5cd" (UID: "0b605f0f-5de4-44c6-9a27-ededc9bbd5cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:55:27.535333 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.535294 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-kube-api-access-pbhd9" pod "0b605f0f-5de4-44c6-9a27-ededc9bbd5cd" (UID: "0b605f0f-5de4-44c6-9a27-ededc9bbd5cd"). InnerVolumeSpecName "kube-api-access-pbhd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:27.535588 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.535568 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-tls-certs" pod "0b605f0f-5de4-44c6-9a27-ededc9bbd5cd" (UID: "0b605f0f-5de4-44c6-9a27-ededc9bbd5cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:55:27.634262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.634191 4335 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-model-cache\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:27.634262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.634232 4335 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-tmp-dir\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:27.634262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.634243 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbhd9\" (UniqueName: \"kubernetes.io/projected/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-kube-api-access-pbhd9\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:27.634262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.634251 4335 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-dshm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:27.634262 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.634260 4335 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd-tls-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:27.845356 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.845312 4335 generic.go:356] "Generic (PLEG): container finished" podID="0b605f0f-5de4-44c6-9a27-ededc9bbd5cd" containerID="0a21f1a9da6b7721f0c86bde0d2175bfe2fc60ae07e96b81230e0c7e46f1eacd" exitCode=0 Oct 30 22:55:27.845517 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.845363 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" event={"ID":"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd","Type":"ContainerDied","Data":"0a21f1a9da6b7721f0c86bde0d2175bfe2fc60ae07e96b81230e0c7e46f1eacd"} Oct 30 22:55:27.845517 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.845380 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" Oct 30 22:55:27.845517 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.845414 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" event={"ID":"0b605f0f-5de4-44c6-9a27-ededc9bbd5cd","Type":"ContainerDied","Data":"2d71219102cd23a28ba49b7e6a20e9a1f163239b04acd7808aad299bb9dde13e"} Oct 30 22:55:27.845517 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.845432 4335 scope.go:122] "RemoveContainer" containerID="0a21f1a9da6b7721f0c86bde0d2175bfe2fc60ae07e96b81230e0c7e46f1eacd" Oct 30 22:55:27.855437 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.855408 4335 scope.go:122] "RemoveContainer" containerID="0a21f1a9da6b7721f0c86bde0d2175bfe2fc60ae07e96b81230e0c7e46f1eacd" Oct 30 22:55:27.855744 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:55:27.855710 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a21f1a9da6b7721f0c86bde0d2175bfe2fc60ae07e96b81230e0c7e46f1eacd\": container with ID starting with 0a21f1a9da6b7721f0c86bde0d2175bfe2fc60ae07e96b81230e0c7e46f1eacd not found: ID does not exist" containerID="0a21f1a9da6b7721f0c86bde0d2175bfe2fc60ae07e96b81230e0c7e46f1eacd" Oct 30 22:55:27.855798 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:27.855757 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a21f1a9da6b7721f0c86bde0d2175bfe2fc60ae07e96b81230e0c7e46f1eacd"} err="failed to get container status \"0a21f1a9da6b7721f0c86bde0d2175bfe2fc60ae07e96b81230e0c7e46f1eacd\": rpc error: code = NotFound desc = could not find container \"0a21f1a9da6b7721f0c86bde0d2175bfe2fc60ae07e96b81230e0c7e46f1eacd\": container with ID starting with 0a21f1a9da6b7721f0c86bde0d2175bfe2fc60ae07e96b81230e0c7e46f1eacd not found: ID does not exist" Oct 30 22:55:28.009643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:28.009571 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572"] Oct 30 22:55:28.013358 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:28.013336 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572"] Oct 30 22:55:28.347449 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:28.347414 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-f5d67e12-kserve-5598c65cc7-46572" podUID="0b605f0f-5de4-44c6-9a27-ededc9bbd5cd" containerName="main" probeResult="failure" output="Get \"https://10.133.1.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:55:29.017794 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:29.017763 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:55:29.056104 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:29.056078 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b605f0f-5de4-44c6-9a27-ededc9bbd5cd" path="/var/lib/kubelet/pods/0b605f0f-5de4-44c6-9a27-ededc9bbd5cd/volumes" Oct 30 22:55:31.239999 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:31.239969 4335 scope.go:122] "RemoveContainer" containerID="b4864d058bb4bb08a0b28fb254322823cd4c69c12b3644b464bb8ee3b667fa5e" Oct 30 22:55:40.356804 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:40.356724 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:55:48.891508 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:48.891470 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg"] Oct 30 22:55:48.892129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:48.892105 4335 cpu_manager.go:414] "containerMap: removing container" podUID="3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af" containerName="revoke-keys" Oct 30 22:55:48.892129 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:48.892130 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af" containerName="revoke-keys" Oct 30 22:55:48.892286 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:48.892155 4335 cpu_manager.go:414] "containerMap: removing container" podUID="902234b4-43b0-4139-9dc3-87543c0c79cf" containerName="main" Oct 30 22:55:48.892286 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:48.892163 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902234b4-43b0-4139-9dc3-87543c0c79cf" containerName="main" Oct 30 22:55:48.892286 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:48.892192 4335 cpu_manager.go:414] "containerMap: removing container" podUID="0b605f0f-5de4-44c6-9a27-ededc9bbd5cd" containerName="main" Oct 30 22:55:48.892286 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:48.892200 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b605f0f-5de4-44c6-9a27-ededc9bbd5cd" containerName="main" Oct 30 22:55:48.892431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:48.892310 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b605f0f-5de4-44c6-9a27-ededc9bbd5cd" containerName="main" Oct 30 22:55:48.892431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:48.892326 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af" containerName="revoke-keys" Oct 30 22:55:48.892431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:48.892334 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="902234b4-43b0-4139-9dc3-87543c0c79cf" containerName="main" Oct 30 22:55:48.895725 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:48.895706 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg" Oct 30 22:55:48.996152 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:48.996119 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c666fa94-x76bg\" (UID: \"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg" Oct 30 22:55:48.996374 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:48.996168 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpldc\" (UniqueName: \"kubernetes.io/projected/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-kube-api-access-hpldc\") pod \"maas-api-revoke-sub-c666fa94-x76bg\" (UID: \"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg" Oct 30 22:55:48.996374 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:48.996330 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c666fa94-x76bg\" (UID: \"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg" Oct 30 22:55:48.997827 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:48.997812 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:55:48.997915 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:48.997899 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:55:48.997969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:48.997905 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:55:48.998156 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:48.998138 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n567f\"" Oct 30 22:55:49.000872 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:49.000853 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg"] Oct 30 22:55:49.096972 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:49.096944 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c666fa94-x76bg\" (UID: \"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg" Oct 30 22:55:49.097114 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:49.096978 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c666fa94-x76bg\" (UID: \"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg" Oct 30 22:55:49.097114 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:49.097016 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpldc\" (UniqueName: \"kubernetes.io/projected/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-kube-api-access-hpldc\") pod \"maas-api-revoke-sub-c666fa94-x76bg\" (UID: \"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg" Oct 30 22:55:49.097654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:49.097606 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c666fa94-x76bg\" (UID: \"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg" Oct 30 22:55:49.099405 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:49.099386 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c666fa94-x76bg\" (UID: \"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg" Oct 30 22:55:49.158377 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:49.158319 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpldc\" (UniqueName: \"kubernetes.io/projected/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-kube-api-access-hpldc\") pod \"maas-api-revoke-sub-c666fa94-x76bg\" (UID: \"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg" Oct 30 22:55:49.208280 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:49.208255 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg" Oct 30 22:55:49.584025 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:49.583997 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg"] Oct 30 22:55:49.584506 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:55:49.584483 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff60efe0_e305_4bd6_bce1_b6b7e5db6f00.slice/crio-f768af26cc6c125abd360359d4c7d07f37c51e15a4d1bc87e85d3c81af94b4d9 WatchSource:0}: Error finding container f768af26cc6c125abd360359d4c7d07f37c51e15a4d1bc87e85d3c81af94b4d9: Status 404 returned error can't find the container with id f768af26cc6c125abd360359d4c7d07f37c51e15a4d1bc87e85d3c81af94b4d9 Oct 30 22:55:50.067297 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:50.067257 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg" event={"ID":"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00","Type":"ContainerStarted","Data":"8b817e15118ef63470ddb76ecce2f7413e7a996f5fd1cea3a58dcd8c1ab1a24d"} Oct 30 22:55:50.067760 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:50.067337 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg" event={"ID":"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00","Type":"ContainerStarted","Data":"f768af26cc6c125abd360359d4c7d07f37c51e15a4d1bc87e85d3c81af94b4d9"} Oct 30 22:55:50.183654 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:50.183540 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg" podStartSLOduration=2.183526983 podStartE2EDuration="2.183526983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:50.181402615 +0000 UTC m=+4283.281834346" watchObservedRunningTime="2026-10-30 22:55:50.183526983 +0000 UTC m=+4283.283958758" Oct 30 22:55:51.074554 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:51.074525 4335 generic.go:356] "Generic (PLEG): container finished" podID="ff60efe0-e305-4bd6-bce1-b6b7e5db6f00" containerID="8b817e15118ef63470ddb76ecce2f7413e7a996f5fd1cea3a58dcd8c1ab1a24d" exitCode=0 Oct 30 22:55:51.074969 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:51.074609 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg" event={"ID":"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00","Type":"ContainerDied","Data":"8b817e15118ef63470ddb76ecce2f7413e7a996f5fd1cea3a58dcd8c1ab1a24d"} Oct 30 22:55:52.215778 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:52.215753 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg" Oct 30 22:55:52.322360 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:52.322335 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-maas-api-openshift-ca\") pod \"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00\" (UID: \"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00\") " Oct 30 22:55:52.322489 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:52.322384 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-kube-api-access-hpldc\" (UniqueName: \"kubernetes.io/projected/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-kube-api-access-hpldc\") pod \"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00\" (UID: \"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00\") " Oct 30 22:55:52.322567 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:52.322499 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-maas-api-xks-ca\") pod \"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00\" (UID: \"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00\") " Oct 30 22:55:52.322767 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:52.322737 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-maas-api-openshift-ca" pod "ff60efe0-e305-4bd6-bce1-b6b7e5db6f00" (UID: "ff60efe0-e305-4bd6-bce1-b6b7e5db6f00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:55:52.322892 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:52.322862 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:52.324627 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:52.324596 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-maas-api-xks-ca" pod "ff60efe0-e305-4bd6-bce1-b6b7e5db6f00" (UID: "ff60efe0-e305-4bd6-bce1-b6b7e5db6f00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:55:52.324765 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:52.324750 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-kube-api-access-hpldc" pod "ff60efe0-e305-4bd6-bce1-b6b7e5db6f00" (UID: "ff60efe0-e305-4bd6-bce1-b6b7e5db6f00"). InnerVolumeSpecName "kube-api-access-hpldc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:52.423707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:52.423654 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:52.423707 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:52.423674 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpldc\" (UniqueName: \"kubernetes.io/projected/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00-kube-api-access-hpldc\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:55:53.086636 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:53.086591 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg" Oct 30 22:55:53.086755 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:53.086639 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg" event={"ID":"ff60efe0-e305-4bd6-bce1-b6b7e5db6f00","Type":"ContainerDied","Data":"f768af26cc6c125abd360359d4c7d07f37c51e15a4d1bc87e85d3c81af94b4d9"} Oct 30 22:55:53.086755 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:53.086686 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f768af26cc6c125abd360359d4c7d07f37c51e15a4d1bc87e85d3c81af94b4d9" Oct 30 22:55:54.104450 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:54.104422 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8"] Oct 30 22:55:54.153902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:54.153878 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vdnq8"] Oct 30 22:55:55.055850 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:55:55.055820 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="279a5123-e61b-4ac0-b664-0716a0fc8968" path="/var/lib/kubelet/pods/279a5123-e61b-4ac0-b664-0716a0fc8968/volumes" Oct 30 22:55:55.320794 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:55:55.320723 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:56:01.716398 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:01.716363 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:56:09.060960 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.060907 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl"] Oct 30 22:56:09.061495 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.061472 4335 cpu_manager.go:414] "containerMap: removing container" podUID="ff60efe0-e305-4bd6-bce1-b6b7e5db6f00" containerName="revoke-keys" Oct 30 22:56:09.061495 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.061492 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff60efe0-e305-4bd6-bce1-b6b7e5db6f00" containerName="revoke-keys" Oct 30 22:56:09.061640 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.061568 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff60efe0-e305-4bd6-bce1-b6b7e5db6f00" containerName="revoke-keys" Oct 30 22:56:09.064850 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.064829 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.117366 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.117335 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a75435c5-openshift-default-dockercfg-jclth\"" Oct 30 22:56:09.170020 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.169989 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl"] Oct 30 22:56:09.177260 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.177226 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc97s\" (UniqueName: \"kubernetes.io/projected/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-kube-api-access-qc97s\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.177412 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.177301 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-token\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.177412 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.177389 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-workload-certs\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.177523 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.177426 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-credential-socket\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.177523 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.177454 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-envoy\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.177593 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.177573 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-workload-socket\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.177662 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.177646 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istiod-ca-cert\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.177741 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.177724 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-data\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.177862 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.177845 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-podinfo\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.278997 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.278963 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc97s\" (UniqueName: \"kubernetes.io/projected/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-kube-api-access-qc97s\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.279209 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.279009 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-token\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.279209 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.279027 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-workload-certs\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.279209 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.279044 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-credential-socket\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.279209 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.279059 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-envoy\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.279209 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.279094 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-workload-socket\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.279209 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.279131 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istiod-ca-cert\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.279209 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.279174 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-data\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.279704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.279245 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-podinfo\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.279704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.279454 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-workload-certs\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.279704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.279527 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-credential-socket\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.279704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.279569 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-workload-socket\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.279704 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.279692 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-data\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.279992 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.279924 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istiod-ca-cert\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.281624 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.281594 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-envoy\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.281688 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.281672 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-podinfo\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.340150 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.340080 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-token\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.340150 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.340130 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc97s\" (UniqueName: \"kubernetes.io/projected/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-kube-api-access-qc97s\") pod \"e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.384166 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.384141 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:09.814109 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.814084 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl"] Oct 30 22:56:09.814694 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:56:09.814668 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d3baf5_761b_43db_b60d_0ef4b35bbbbf.slice/crio-7b9155a3ebc9be1464bebf9cc0e55a58d6d45575dcd5d80924cabb0bc0234ca8 WatchSource:0}: Error finding container 7b9155a3ebc9be1464bebf9cc0e55a58d6d45575dcd5d80924cabb0bc0234ca8: Status 404 returned error can't find the container with id 7b9155a3ebc9be1464bebf9cc0e55a58d6d45575dcd5d80924cabb0bc0234ca8 Oct 30 22:56:09.816726 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.816691 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:56:09.816851 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.816758 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:56:09.816851 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:09.816784 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:56:10.187902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:10.187816 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" event={"ID":"36d3baf5-761b-43db-b60d-0ef4b35bbbbf","Type":"ContainerStarted","Data":"32fed0cc4ac14c2b11f052073bcd79860570ceab4b630a4adc595137ba255e19"} Oct 30 22:56:10.187902 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:10.187873 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" event={"ID":"36d3baf5-761b-43db-b60d-0ef4b35bbbbf","Type":"ContainerStarted","Data":"7b9155a3ebc9be1464bebf9cc0e55a58d6d45575dcd5d80924cabb0bc0234ca8"} Oct 30 22:56:10.357378 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:10.357331 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" podStartSLOduration=1.357312616 podStartE2EDuration="1.357312616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:10.356762381 +0000 UTC m=+4303.457194114" watchObservedRunningTime="2026-10-30 22:56:10.357312616 +0000 UTC m=+4303.457744349" Oct 30 22:56:10.384831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:10.384805 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:10.390037 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:10.390016 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:11.194413 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:11.194388 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:11.195354 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:11.195333 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:16.522073 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:16.522044 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx"] Oct 30 22:56:16.525796 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:16.525779 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx" Oct 30 22:56:16.579242 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:16.579218 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a75435c5\"" Oct 30 22:56:16.628533 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:16.628512 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a75435c5-dockercfg-m8zg8\"" Oct 30 22:56:16.631568 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:16.631546 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx"] Oct 30 22:56:16.653005 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:16.652978 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1811b7-d399-40f5-86f4-02dbc64e0c17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx\" (UID: \"8e1811b7-d399-40f5-86f4-02dbc64e0c17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx" Oct 30 22:56:16.653115 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:16.653019 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6txw\" (UniqueName: \"kubernetes.io/projected/8e1811b7-d399-40f5-86f4-02dbc64e0c17-kube-api-access-q6txw\") pod \"payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx\" (UID: \"8e1811b7-d399-40f5-86f4-02dbc64e0c17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx" Oct 30 22:56:16.754400 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:16.754378 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1811b7-d399-40f5-86f4-02dbc64e0c17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx\" (UID: \"8e1811b7-d399-40f5-86f4-02dbc64e0c17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx" Oct 30 22:56:16.754500 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:16.754415 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6txw\" (UniqueName: \"kubernetes.io/projected/8e1811b7-d399-40f5-86f4-02dbc64e0c17-kube-api-access-q6txw\") pod \"payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx\" (UID: \"8e1811b7-d399-40f5-86f4-02dbc64e0c17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx" Oct 30 22:56:16.755078 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:16.755062 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1811b7-d399-40f5-86f4-02dbc64e0c17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx\" (UID: \"8e1811b7-d399-40f5-86f4-02dbc64e0c17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx" Oct 30 22:56:16.814606 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:16.814578 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6txw\" (UniqueName: \"kubernetes.io/projected/8e1811b7-d399-40f5-86f4-02dbc64e0c17-kube-api-access-q6txw\") pod \"payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx\" (UID: \"8e1811b7-d399-40f5-86f4-02dbc64e0c17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx" Oct 30 22:56:16.839444 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:16.839422 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx" Oct 30 22:56:17.214757 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:17.214729 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx"] Oct 30 22:56:17.216159 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:56:17.216127 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1811b7_d399_40f5_86f4_02dbc64e0c17.slice/crio-b32c07fe03bbe42aa12f2808e325bc0ac65c9add14c11913e2d38c0988c81d62 WatchSource:0}: Error finding container b32c07fe03bbe42aa12f2808e325bc0ac65c9add14c11913e2d38c0988c81d62: Status 404 returned error can't find the container with id b32c07fe03bbe42aa12f2808e325bc0ac65c9add14c11913e2d38c0988c81d62 Oct 30 22:56:17.235771 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:17.235746 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx" event={"ID":"8e1811b7-d399-40f5-86f4-02dbc64e0c17","Type":"ContainerStarted","Data":"b32c07fe03bbe42aa12f2808e325bc0ac65c9add14c11913e2d38c0988c81d62"} Oct 30 22:56:18.244749 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:18.244716 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx" event={"ID":"8e1811b7-d399-40f5-86f4-02dbc64e0c17","Type":"ContainerStarted","Data":"400b2baa811ebbb9684776fdc87c1f358857b35b55bd6debd421caca38942411"} Oct 30 22:56:18.245175 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:18.244780 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx" Oct 30 22:56:18.361580 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:18.361532 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx" podStartSLOduration=2.361517996 podStartE2EDuration="2.361517996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:18.359343674 +0000 UTC m=+4311.459775402" watchObservedRunningTime="2026-10-30 22:56:18.361517996 +0000 UTC m=+4311.461949768" Oct 30 22:56:24.049422 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:24.049383 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q"] Oct 30 22:56:24.098281 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:24.098255 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-s5n8q"] Oct 30 22:56:25.056214 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:25.056180 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="949ecb4d-efd6-4b82-af45-56f8d7e33d00" path="/var/lib/kubelet/pods/949ecb4d-efd6-4b82-af45-56f8d7e33d00/volumes" Oct 30 22:56:25.320624 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:56:25.320530 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:56:26.123943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:26.123905 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:56:29.253698 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:29.253666 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx" Oct 30 22:56:35.387328 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:35.387294 4335 generic.go:356] "Generic (PLEG): container finished" podID="36d3baf5-761b-43db-b60d-0ef4b35bbbbf" containerID="32fed0cc4ac14c2b11f052073bcd79860570ceab4b630a4adc595137ba255e19" exitCode=0 Oct 30 22:56:35.387819 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:35.387365 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" event={"ID":"36d3baf5-761b-43db-b60d-0ef4b35bbbbf","Type":"ContainerDied","Data":"32fed0cc4ac14c2b11f052073bcd79860570ceab4b630a4adc595137ba255e19"} Oct 30 22:56:35.387819 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:35.387724 4335 scope.go:122] "RemoveContainer" containerID="32fed0cc4ac14c2b11f052073bcd79860570ceab4b630a4adc595137ba255e19" Oct 30 22:56:35.388897 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:35.388865 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:56:35.389013 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:35.388941 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:56:35.389013 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:35.388976 4335 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:56:36.395843 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:36.395808 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" event={"ID":"36d3baf5-761b-43db-b60d-0ef4b35bbbbf","Type":"ContainerStarted","Data":"30b33949a3a6fecfb36660bbcc3949c5cbf7c3dcc084205cdf143ac4d35d2c20"} Oct 30 22:56:37.384717 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:37.384685 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:37.385677 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:37.385645 4335 patch_prober.go:29] interesting pod/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Oct 30 22:56:37.385815 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:37.385704 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" podUID="36d3baf5-761b-43db-b60d-0ef4b35bbbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Oct 30 22:56:38.385526 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:38.385488 4335 patch_prober.go:29] interesting pod/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Oct 30 22:56:38.386023 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:38.385553 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" podUID="36d3baf5-761b-43db-b60d-0ef4b35bbbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Oct 30 22:56:39.384942 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:39.384912 4335 patch_prober.go:29] interesting pod/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Oct 30 22:56:39.385137 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:39.384973 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" podUID="36d3baf5-761b-43db-b60d-0ef4b35bbbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Oct 30 22:56:40.384860 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:40.384824 4335 patch_prober.go:29] interesting pod/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Oct 30 22:56:40.385302 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:40.384888 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" podUID="36d3baf5-761b-43db-b60d-0ef4b35bbbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Oct 30 22:56:41.195217 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:41.195192 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:41.385111 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:41.385076 4335 patch_prober.go:29] interesting pod/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Oct 30 22:56:41.385530 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:41.385134 4335 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" podUID="36d3baf5-761b-43db-b60d-0ef4b35bbbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Oct 30 22:56:42.388456 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:42.388428 4335 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:42.389482 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:42.389460 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:56:48.906150 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:48.906113 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5"] Oct 30 22:56:48.908721 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:48.908705 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5" Oct 30 22:56:48.958985 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:48.958964 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4570e76e-c85e-4b9f-9936-2ef708bc8624-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5\" (UID: \"4570e76e-c85e-4b9f-9936-2ef708bc8624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5" Oct 30 22:56:48.959117 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:48.959006 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4570e76e-c85e-4b9f-9936-2ef708bc8624-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5\" (UID: \"4570e76e-c85e-4b9f-9936-2ef708bc8624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5" Oct 30 22:56:48.959117 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:48.959036 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjtg\" (UniqueName: \"kubernetes.io/projected/4570e76e-c85e-4b9f-9936-2ef708bc8624-kube-api-access-6xjtg\") pod \"maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5\" (UID: \"4570e76e-c85e-4b9f-9936-2ef708bc8624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5" Oct 30 22:56:48.963601 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:48.963571 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:56:49.012846 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:49.012822 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:56:49.012966 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:49.012850 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:56:49.012966 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:49.012952 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvx84\"" Oct 30 22:56:49.016033 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:49.016011 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5"] Oct 30 22:56:49.059565 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:49.059543 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4570e76e-c85e-4b9f-9936-2ef708bc8624-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5\" (UID: \"4570e76e-c85e-4b9f-9936-2ef708bc8624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5" Oct 30 22:56:49.059706 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:49.059588 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4570e76e-c85e-4b9f-9936-2ef708bc8624-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5\" (UID: \"4570e76e-c85e-4b9f-9936-2ef708bc8624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5" Oct 30 22:56:49.059706 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:49.059644 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjtg\" (UniqueName: \"kubernetes.io/projected/4570e76e-c85e-4b9f-9936-2ef708bc8624-kube-api-access-6xjtg\") pod \"maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5\" (UID: \"4570e76e-c85e-4b9f-9936-2ef708bc8624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5" Oct 30 22:56:49.060206 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:49.060176 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4570e76e-c85e-4b9f-9936-2ef708bc8624-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5\" (UID: \"4570e76e-c85e-4b9f-9936-2ef708bc8624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5" Oct 30 22:56:49.062051 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:49.062033 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4570e76e-c85e-4b9f-9936-2ef708bc8624-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5\" (UID: \"4570e76e-c85e-4b9f-9936-2ef708bc8624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5" Oct 30 22:56:49.120786 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:49.120760 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjtg\" (UniqueName: \"kubernetes.io/projected/4570e76e-c85e-4b9f-9936-2ef708bc8624-kube-api-access-6xjtg\") pod \"maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5\" (UID: \"4570e76e-c85e-4b9f-9936-2ef708bc8624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5" Oct 30 22:56:49.220466 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:49.220416 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5" Oct 30 22:56:49.594554 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:49.594528 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5"] Oct 30 22:56:49.596161 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:56:49.596132 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4570e76e_c85e_4b9f_9936_2ef708bc8624.slice/crio-41805e23f5011aec967c0352cef8413f0bd389941834e47b46adef511253496c WatchSource:0}: Error finding container 41805e23f5011aec967c0352cef8413f0bd389941834e47b46adef511253496c: Status 404 returned error can't find the container with id 41805e23f5011aec967c0352cef8413f0bd389941834e47b46adef511253496c Oct 30 22:56:50.508978 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:50.508940 4335 generic.go:356] "Generic (PLEG): container finished" podID="4570e76e-c85e-4b9f-9936-2ef708bc8624" containerID="d285635ea3b2620be56fae6f14150e8ff55e5d51d8a2fcfcf7b3415ae9b0e42b" exitCode=0 Oct 30 22:56:50.509303 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:50.509022 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5" event={"ID":"4570e76e-c85e-4b9f-9936-2ef708bc8624","Type":"ContainerDied","Data":"d285635ea3b2620be56fae6f14150e8ff55e5d51d8a2fcfcf7b3415ae9b0e42b"} Oct 30 22:56:50.509303 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:50.509071 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5" event={"ID":"4570e76e-c85e-4b9f-9936-2ef708bc8624","Type":"ContainerStarted","Data":"41805e23f5011aec967c0352cef8413f0bd389941834e47b46adef511253496c"} Oct 30 22:56:51.640289 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:51.640267 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5" Oct 30 22:56:51.784841 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:51.784766 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4570e76e-c85e-4b9f-9936-2ef708bc8624-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4570e76e-c85e-4b9f-9936-2ef708bc8624-maas-api-openshift-ca\") pod \"4570e76e-c85e-4b9f-9936-2ef708bc8624\" (UID: \"4570e76e-c85e-4b9f-9936-2ef708bc8624\") " Oct 30 22:56:51.784841 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:51.784839 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4570e76e-c85e-4b9f-9936-2ef708bc8624-kube-api-access-6xjtg\" (UniqueName: \"kubernetes.io/projected/4570e76e-c85e-4b9f-9936-2ef708bc8624-kube-api-access-6xjtg\") pod \"4570e76e-c85e-4b9f-9936-2ef708bc8624\" (UID: \"4570e76e-c85e-4b9f-9936-2ef708bc8624\") " Oct 30 22:56:51.785069 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:51.784874 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4570e76e-c85e-4b9f-9936-2ef708bc8624-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4570e76e-c85e-4b9f-9936-2ef708bc8624-maas-api-xks-ca\") pod \"4570e76e-c85e-4b9f-9936-2ef708bc8624\" (UID: \"4570e76e-c85e-4b9f-9936-2ef708bc8624\") " Oct 30 22:56:51.785137 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:51.785087 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4570e76e-c85e-4b9f-9936-2ef708bc8624-maas-api-openshift-ca" pod "4570e76e-c85e-4b9f-9936-2ef708bc8624" (UID: "4570e76e-c85e-4b9f-9936-2ef708bc8624"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:56:51.785237 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:51.785220 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4570e76e-c85e-4b9f-9936-2ef708bc8624-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:51.787026 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:51.787005 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4570e76e-c85e-4b9f-9936-2ef708bc8624-kube-api-access-6xjtg" pod "4570e76e-c85e-4b9f-9936-2ef708bc8624" (UID: "4570e76e-c85e-4b9f-9936-2ef708bc8624"). InnerVolumeSpecName "kube-api-access-6xjtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:51.787144 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:51.787043 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4570e76e-c85e-4b9f-9936-2ef708bc8624-maas-api-xks-ca" pod "4570e76e-c85e-4b9f-9936-2ef708bc8624" (UID: "4570e76e-c85e-4b9f-9936-2ef708bc8624"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:56:51.885840 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:51.885816 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xjtg\" (UniqueName: \"kubernetes.io/projected/4570e76e-c85e-4b9f-9936-2ef708bc8624-kube-api-access-6xjtg\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:51.885840 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:51.885839 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4570e76e-c85e-4b9f-9936-2ef708bc8624-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:56:52.523225 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:52.523189 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5" event={"ID":"4570e76e-c85e-4b9f-9936-2ef708bc8624","Type":"ContainerDied","Data":"41805e23f5011aec967c0352cef8413f0bd389941834e47b46adef511253496c"} Oct 30 22:56:52.523225 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:52.523226 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5" Oct 30 22:56:52.523431 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:52.523252 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41805e23f5011aec967c0352cef8413f0bd389941834e47b46adef511253496c" Oct 30 22:56:55.321469 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:56:55.321425 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:56:59.397361 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:56:59.397329 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx"] Oct 30 22:57:04.398215 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:04.398173 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx" podUID="8e1811b7-d399-40f5-86f4-02dbc64e0c17" containerName="payload-pre-processing" containerID="cri-o://400b2baa811ebbb9684776fdc87c1f358857b35b55bd6debd421caca38942411" gracePeriod=25 Oct 30 22:57:04.607356 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:04.607314 4335 generic.go:356] "Generic (PLEG): container finished" podID="8e1811b7-d399-40f5-86f4-02dbc64e0c17" containerID="400b2baa811ebbb9684776fdc87c1f358857b35b55bd6debd421caca38942411" exitCode=0 Oct 30 22:57:04.607532 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:04.607392 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx" event={"ID":"8e1811b7-d399-40f5-86f4-02dbc64e0c17","Type":"ContainerDied","Data":"400b2baa811ebbb9684776fdc87c1f358857b35b55bd6debd421caca38942411"} Oct 30 22:57:04.647211 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:04.647190 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx" Oct 30 22:57:04.799977 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:04.799953 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1811b7-d399-40f5-86f4-02dbc64e0c17-kube-api-access-q6txw\" (UniqueName: \"kubernetes.io/projected/8e1811b7-d399-40f5-86f4-02dbc64e0c17-kube-api-access-q6txw\") pod \"8e1811b7-d399-40f5-86f4-02dbc64e0c17\" (UID: \"8e1811b7-d399-40f5-86f4-02dbc64e0c17\") " Oct 30 22:57:04.800133 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:04.800032 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e1811b7-d399-40f5-86f4-02dbc64e0c17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1811b7-d399-40f5-86f4-02dbc64e0c17-plugins-config-volume\") pod \"8e1811b7-d399-40f5-86f4-02dbc64e0c17\" (UID: \"8e1811b7-d399-40f5-86f4-02dbc64e0c17\") " Oct 30 22:57:04.800473 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:04.800446 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1811b7-d399-40f5-86f4-02dbc64e0c17-plugins-config-volume" pod "8e1811b7-d399-40f5-86f4-02dbc64e0c17" (UID: "8e1811b7-d399-40f5-86f4-02dbc64e0c17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:57:04.802229 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:04.802205 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1811b7-d399-40f5-86f4-02dbc64e0c17-kube-api-access-q6txw" pod "8e1811b7-d399-40f5-86f4-02dbc64e0c17" (UID: "8e1811b7-d399-40f5-86f4-02dbc64e0c17"). InnerVolumeSpecName "kube-api-access-q6txw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:57:04.901209 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:04.901185 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1811b7-d399-40f5-86f4-02dbc64e0c17-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:04.901209 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:04.901207 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6txw\" (UniqueName: \"kubernetes.io/projected/8e1811b7-d399-40f5-86f4-02dbc64e0c17-kube-api-access-q6txw\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:05.614474 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:05.614445 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx" event={"ID":"8e1811b7-d399-40f5-86f4-02dbc64e0c17","Type":"ContainerDied","Data":"b32c07fe03bbe42aa12f2808e325bc0ac65c9add14c11913e2d38c0988c81d62"} Oct 30 22:57:05.614474 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:05.614469 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx" Oct 30 22:57:05.615076 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:05.614499 4335 scope.go:122] "RemoveContainer" containerID="400b2baa811ebbb9684776fdc87c1f358857b35b55bd6debd421caca38942411" Oct 30 22:57:05.780088 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:05.780061 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx"] Oct 30 22:57:05.829034 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:05.829010 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a75435c5-7c795f7b5f-gv9hx"] Oct 30 22:57:07.056353 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:07.056323 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e1811b7-d399-40f5-86f4-02dbc64e0c17" path="/var/lib/kubelet/pods/8e1811b7-d399-40f5-86f4-02dbc64e0c17/volumes" Oct 30 22:57:09.704284 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:09.704206 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5"] Oct 30 22:57:09.753490 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:09.753466 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a75435c5-fv8q5"] Oct 30 22:57:11.056450 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:11.056415 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4570e76e-c85e-4b9f-9936-2ef708bc8624" path="/var/lib/kubelet/pods/4570e76e-c85e-4b9f-9936-2ef708bc8624/volumes" Oct 30 22:57:12.156962 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:12.156930 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:57:15.030767 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:15.030733 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl"] Oct 30 22:57:15.031172 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:15.030963 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" podUID="36d3baf5-761b-43db-b60d-0ef4b35bbbbf" containerName="istio-proxy" containerID="cri-o://30b33949a3a6fecfb36660bbcc3949c5cbf7c3dcc084205cdf143ac4d35d2c20" gracePeriod=30 Oct 30 22:57:20.275912 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.275888 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:57:20.339943 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.339915 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istiod-ca-cert\") pod \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " Oct 30 22:57:20.340082 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.339953 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-token\" (UniqueName: \"kubernetes.io/projected/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-token\") pod \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " Oct 30 22:57:20.340082 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.340009 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-workload-certs\") pod \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " Oct 30 22:57:20.340082 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.340036 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-workload-socket\") pod \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " Oct 30 22:57:20.340082 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.340055 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-data\") pod \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " Oct 30 22:57:20.340308 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.340084 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-credential-socket\") pod \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " Oct 30 22:57:20.340308 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.340126 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-podinfo\") pod \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " Oct 30 22:57:20.340308 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.340220 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-kube-api-access-qc97s\" (UniqueName: \"kubernetes.io/projected/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-kube-api-access-qc97s\") pod \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " Oct 30 22:57:20.340308 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.340258 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-envoy\") pod \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\" (UID: \"36d3baf5-761b-43db-b60d-0ef4b35bbbbf\") " Oct 30 22:57:20.340544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.340325 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-workload-certs" pod "36d3baf5-761b-43db-b60d-0ef4b35bbbbf" (UID: "36d3baf5-761b-43db-b60d-0ef4b35bbbbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:57:20.340544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.340415 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-data" pod "36d3baf5-761b-43db-b60d-0ef4b35bbbbf" (UID: "36d3baf5-761b-43db-b60d-0ef4b35bbbbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:57:20.340544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.340317 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istiod-ca-cert" pod "36d3baf5-761b-43db-b60d-0ef4b35bbbbf" (UID: "36d3baf5-761b-43db-b60d-0ef4b35bbbbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:57:20.340544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.340361 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-workload-socket" pod "36d3baf5-761b-43db-b60d-0ef4b35bbbbf" (UID: "36d3baf5-761b-43db-b60d-0ef4b35bbbbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:57:20.340792 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.340543 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-credential-socket" pod "36d3baf5-761b-43db-b60d-0ef4b35bbbbf" (UID: "36d3baf5-761b-43db-b60d-0ef4b35bbbbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:57:20.340792 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.340741 4335 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istiod-ca-cert\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:20.340792 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.340763 4335 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-workload-certs\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:20.340792 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.340778 4335 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-workload-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:20.340792 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.340792 4335 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-data\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:20.341068 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.340805 4335 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-credential-socket\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:20.342558 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.342532 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-token" pod "36d3baf5-761b-43db-b60d-0ef4b35bbbbf" (UID: "36d3baf5-761b-43db-b60d-0ef4b35bbbbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:57:20.342756 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.342663 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-envoy" pod "36d3baf5-761b-43db-b60d-0ef4b35bbbbf" (UID: "36d3baf5-761b-43db-b60d-0ef4b35bbbbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:57:20.343132 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.343104 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-podinfo" pod "36d3baf5-761b-43db-b60d-0ef4b35bbbbf" (UID: "36d3baf5-761b-43db-b60d-0ef4b35bbbbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:57:20.343185 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.343136 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-kube-api-access-qc97s" pod "36d3baf5-761b-43db-b60d-0ef4b35bbbbf" (UID: "36d3baf5-761b-43db-b60d-0ef4b35bbbbf"). InnerVolumeSpecName "kube-api-access-qc97s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:57:20.441682 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.441653 4335 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-envoy\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:20.441682 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.441675 4335 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-token\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:20.441682 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.441685 4335 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-istio-podinfo\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:20.441955 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.441696 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc97s\" (UniqueName: \"kubernetes.io/projected/36d3baf5-761b-43db-b60d-0ef4b35bbbbf-kube-api-access-qc97s\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:57:20.711068 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.710994 4335 generic.go:356] "Generic (PLEG): container finished" podID="36d3baf5-761b-43db-b60d-0ef4b35bbbbf" containerID="30b33949a3a6fecfb36660bbcc3949c5cbf7c3dcc084205cdf143ac4d35d2c20" exitCode=0 Oct 30 22:57:20.711068 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.711048 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" Oct 30 22:57:20.711249 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.711073 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" event={"ID":"36d3baf5-761b-43db-b60d-0ef4b35bbbbf","Type":"ContainerDied","Data":"30b33949a3a6fecfb36660bbcc3949c5cbf7c3dcc084205cdf143ac4d35d2c20"} Oct 30 22:57:20.711249 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.711125 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl" event={"ID":"36d3baf5-761b-43db-b60d-0ef4b35bbbbf","Type":"ContainerDied","Data":"7b9155a3ebc9be1464bebf9cc0e55a58d6d45575dcd5d80924cabb0bc0234ca8"} Oct 30 22:57:20.711249 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.711147 4335 scope.go:122] "RemoveContainer" containerID="30b33949a3a6fecfb36660bbcc3949c5cbf7c3dcc084205cdf143ac4d35d2c20" Oct 30 22:57:20.721840 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.721823 4335 scope.go:122] "RemoveContainer" containerID="32fed0cc4ac14c2b11f052073bcd79860570ceab4b630a4adc595137ba255e19" Oct 30 22:57:20.730536 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.730519 4335 scope.go:122] "RemoveContainer" containerID="30b33949a3a6fecfb36660bbcc3949c5cbf7c3dcc084205cdf143ac4d35d2c20" Oct 30 22:57:20.730827 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:57:20.730807 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30b33949a3a6fecfb36660bbcc3949c5cbf7c3dcc084205cdf143ac4d35d2c20\": container with ID starting with 30b33949a3a6fecfb36660bbcc3949c5cbf7c3dcc084205cdf143ac4d35d2c20 not found: ID does not exist" containerID="30b33949a3a6fecfb36660bbcc3949c5cbf7c3dcc084205cdf143ac4d35d2c20" Oct 30 22:57:20.730904 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.730839 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30b33949a3a6fecfb36660bbcc3949c5cbf7c3dcc084205cdf143ac4d35d2c20"} err="failed to get container status \"30b33949a3a6fecfb36660bbcc3949c5cbf7c3dcc084205cdf143ac4d35d2c20\": rpc error: code = NotFound desc = could not find container \"30b33949a3a6fecfb36660bbcc3949c5cbf7c3dcc084205cdf143ac4d35d2c20\": container with ID starting with 30b33949a3a6fecfb36660bbcc3949c5cbf7c3dcc084205cdf143ac4d35d2c20 not found: ID does not exist" Oct 30 22:57:20.730904 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.730862 4335 scope.go:122] "RemoveContainer" containerID="32fed0cc4ac14c2b11f052073bcd79860570ceab4b630a4adc595137ba255e19" Oct 30 22:57:20.731099 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:57:20.731082 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32fed0cc4ac14c2b11f052073bcd79860570ceab4b630a4adc595137ba255e19\": container with ID starting with 32fed0cc4ac14c2b11f052073bcd79860570ceab4b630a4adc595137ba255e19 not found: ID does not exist" containerID="32fed0cc4ac14c2b11f052073bcd79860570ceab4b630a4adc595137ba255e19" Oct 30 22:57:20.731146 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.731108 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32fed0cc4ac14c2b11f052073bcd79860570ceab4b630a4adc595137ba255e19"} err="failed to get container status \"32fed0cc4ac14c2b11f052073bcd79860570ceab4b630a4adc595137ba255e19\": rpc error: code = NotFound desc = could not find container \"32fed0cc4ac14c2b11f052073bcd79860570ceab4b630a4adc595137ba255e19\": container with ID starting with 32fed0cc4ac14c2b11f052073bcd79860570ceab4b630a4adc595137ba255e19 not found: ID does not exist" Oct 30 22:57:20.876405 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.876378 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl"] Oct 30 22:57:20.925200 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:20.925178 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a75435c5-openshift-default-5bcb76f559-qvwtl"] Oct 30 22:57:21.056196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:21.056169 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36d3baf5-761b-43db-b60d-0ef4b35bbbbf" path="/var/lib/kubelet/pods/36d3baf5-761b-43db-b60d-0ef4b35bbbbf/volumes" Oct 30 22:57:24.607253 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.607212 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7"] Oct 30 22:57:24.607793 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.607758 4335 cpu_manager.go:414] "containerMap: removing container" podUID="36d3baf5-761b-43db-b60d-0ef4b35bbbbf" containerName="istio-proxy" Oct 30 22:57:24.607793 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.607774 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d3baf5-761b-43db-b60d-0ef4b35bbbbf" containerName="istio-proxy" Oct 30 22:57:24.607917 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.607798 4335 cpu_manager.go:414] "containerMap: removing container" podUID="4570e76e-c85e-4b9f-9936-2ef708bc8624" containerName="revoke-keys" Oct 30 22:57:24.607917 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.607803 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4570e76e-c85e-4b9f-9936-2ef708bc8624" containerName="revoke-keys" Oct 30 22:57:24.607917 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.607816 4335 cpu_manager.go:414] "containerMap: removing container" podUID="36d3baf5-761b-43db-b60d-0ef4b35bbbbf" containerName="istio-proxy" Oct 30 22:57:24.607917 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.607821 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d3baf5-761b-43db-b60d-0ef4b35bbbbf" containerName="istio-proxy" Oct 30 22:57:24.607917 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.607832 4335 cpu_manager.go:414] "containerMap: removing container" podUID="8e1811b7-d399-40f5-86f4-02dbc64e0c17" containerName="payload-pre-processing" Oct 30 22:57:24.607917 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.607836 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1811b7-d399-40f5-86f4-02dbc64e0c17" containerName="payload-pre-processing" Oct 30 22:57:24.607917 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.607905 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d3baf5-761b-43db-b60d-0ef4b35bbbbf" containerName="istio-proxy" Oct 30 22:57:24.607917 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.607916 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="4570e76e-c85e-4b9f-9936-2ef708bc8624" containerName="revoke-keys" Oct 30 22:57:24.608346 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.607923 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1811b7-d399-40f5-86f4-02dbc64e0c17" containerName="payload-pre-processing" Oct 30 22:57:24.613011 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.612984 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7" Oct 30 22:57:24.665515 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.665487 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7"] Oct 30 22:57:24.676196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.676173 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5xdf\" (UniqueName: \"kubernetes.io/projected/1635fc97-dc4b-4e6a-a842-39f39135d803-kube-api-access-r5xdf\") pod \"payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7\" (UID: \"1635fc97-dc4b-4e6a-a842-39f39135d803\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7" Oct 30 22:57:24.676360 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.676230 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1635fc97-dc4b-4e6a-a842-39f39135d803-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7\" (UID: \"1635fc97-dc4b-4e6a-a842-39f39135d803\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7" Oct 30 22:57:24.711721 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.711699 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-8f0848\"" Oct 30 22:57:24.711838 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.711747 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-8f0848-dockercfg-cchq8\"" Oct 30 22:57:24.774553 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.774523 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp"] Oct 30 22:57:24.775255 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.775233 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d3baf5-761b-43db-b60d-0ef4b35bbbbf" containerName="istio-proxy" Oct 30 22:57:24.776803 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.776785 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5xdf\" (UniqueName: \"kubernetes.io/projected/1635fc97-dc4b-4e6a-a842-39f39135d803-kube-api-access-r5xdf\") pod \"payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7\" (UID: \"1635fc97-dc4b-4e6a-a842-39f39135d803\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7" Oct 30 22:57:24.776897 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.776839 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1635fc97-dc4b-4e6a-a842-39f39135d803-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7\" (UID: \"1635fc97-dc4b-4e6a-a842-39f39135d803\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7" Oct 30 22:57:24.777498 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.777480 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1635fc97-dc4b-4e6a-a842-39f39135d803-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7\" (UID: \"1635fc97-dc4b-4e6a-a842-39f39135d803\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7" Oct 30 22:57:24.793950 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.793931 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp" Oct 30 22:57:24.837271 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.837241 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp"] Oct 30 22:57:24.839983 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.839956 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5xdf\" (UniqueName: \"kubernetes.io/projected/1635fc97-dc4b-4e6a-a842-39f39135d803-kube-api-access-r5xdf\") pod \"payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7\" (UID: \"1635fc97-dc4b-4e6a-a842-39f39135d803\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7" Oct 30 22:57:24.877700 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.877604 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a86627d3-cce0-41b1-9e88-12bffcdd0a3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp\" (UID: \"a86627d3-cce0-41b1-9e88-12bffcdd0a3c\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp" Oct 30 22:57:24.877812 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.877741 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hk4p\" (UniqueName: \"kubernetes.io/projected/a86627d3-cce0-41b1-9e88-12bffcdd0a3c-kube-api-access-8hk4p\") pod \"payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp\" (UID: \"a86627d3-cce0-41b1-9e88-12bffcdd0a3c\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp" Oct 30 22:57:24.925935 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.925912 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7" Oct 30 22:57:24.978839 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.978807 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hk4p\" (UniqueName: \"kubernetes.io/projected/a86627d3-cce0-41b1-9e88-12bffcdd0a3c-kube-api-access-8hk4p\") pod \"payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp\" (UID: \"a86627d3-cce0-41b1-9e88-12bffcdd0a3c\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp" Oct 30 22:57:24.979030 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.978890 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a86627d3-cce0-41b1-9e88-12bffcdd0a3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp\" (UID: \"a86627d3-cce0-41b1-9e88-12bffcdd0a3c\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp" Oct 30 22:57:24.979544 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:24.979523 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a86627d3-cce0-41b1-9e88-12bffcdd0a3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp\" (UID: \"a86627d3-cce0-41b1-9e88-12bffcdd0a3c\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp" Oct 30 22:57:25.039823 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:25.039787 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hk4p\" (UniqueName: \"kubernetes.io/projected/a86627d3-cce0-41b1-9e88-12bffcdd0a3c-kube-api-access-8hk4p\") pod \"payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp\" (UID: \"a86627d3-cce0-41b1-9e88-12bffcdd0a3c\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp" Oct 30 22:57:25.107392 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:25.107360 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp" Oct 30 22:57:25.308510 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:25.308471 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7"] Oct 30 22:57:25.310059 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:57:25.310032 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1635fc97_dc4b_4e6a_a842_39f39135d803.slice/crio-8c1376812603e77b8c3627301c783964521368cf22411cd4741204e63eca228a WatchSource:0}: Error finding container 8c1376812603e77b8c3627301c783964521368cf22411cd4741204e63eca228a: Status 404 returned error can't find the container with id 8c1376812603e77b8c3627301c783964521368cf22411cd4741204e63eca228a Oct 30 22:57:25.321670 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:57:25.321645 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:57:25.484985 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:25.484958 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp"] Oct 30 22:57:25.486254 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:57:25.486227 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86627d3_cce0_41b1_9e88_12bffcdd0a3c.slice/crio-31bd2229a142b9e0769711993ba750452679dd9ef5524b91f395fb305b632f1d WatchSource:0}: Error finding container 31bd2229a142b9e0769711993ba750452679dd9ef5524b91f395fb305b632f1d: Status 404 returned error can't find the container with id 31bd2229a142b9e0769711993ba750452679dd9ef5524b91f395fb305b632f1d Oct 30 22:57:25.743825 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:25.743753 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7" event={"ID":"1635fc97-dc4b-4e6a-a842-39f39135d803","Type":"ContainerStarted","Data":"2e90e614d5e09aa5f59602b891a59eadd56be4705e98b33ecbf4096fb4026563"} Oct 30 22:57:25.743825 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:25.743817 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7" event={"ID":"1635fc97-dc4b-4e6a-a842-39f39135d803","Type":"ContainerStarted","Data":"8c1376812603e77b8c3627301c783964521368cf22411cd4741204e63eca228a"} Oct 30 22:57:25.744232 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:25.743849 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7" Oct 30 22:57:25.745407 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:25.745384 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp" event={"ID":"a86627d3-cce0-41b1-9e88-12bffcdd0a3c","Type":"ContainerStarted","Data":"0e80c953ba2e7f5ca423631a778e62524ba2935185613d4b4b929cec45d27991"} Oct 30 22:57:25.745522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:25.745430 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp" event={"ID":"a86627d3-cce0-41b1-9e88-12bffcdd0a3c","Type":"ContainerStarted","Data":"31bd2229a142b9e0769711993ba750452679dd9ef5524b91f395fb305b632f1d"} Oct 30 22:57:25.745522 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:25.745470 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp" Oct 30 22:57:25.862182 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:25.862136 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7" podStartSLOduration=1.862124417 podStartE2EDuration="1.862124417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:25.85915669 +0000 UTC m=+4378.959588424" watchObservedRunningTime="2026-10-30 22:57:25.862124417 +0000 UTC m=+4378.962556187" Oct 30 22:57:25.977273 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:25.977202 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp" podStartSLOduration=1.977183108 podStartE2EDuration="1.977183108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:25.974123923 +0000 UTC m=+4379.074555656" watchObservedRunningTime="2026-10-30 22:57:25.977183108 +0000 UTC m=+4379.077614841" Oct 30 22:57:31.360797 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:31.360767 4335 scope.go:122] "RemoveContainer" containerID="1b6c3bf0b03ee4f281ccec0fa70d6ac66deb4d99eb9cf972b2baee0455ccac1d" Oct 30 22:57:31.370186 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:31.370169 4335 scope.go:122] "RemoveContainer" containerID="4a198847a050ba2a2c6dffd8eb5a2398261b2c53e39ee78e86fa8d5a5f93277b" Oct 30 22:57:35.230549 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:35.230515 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:57:36.754286 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:36.754261 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7" Oct 30 22:57:36.754789 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:36.754324 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp" Oct 30 22:57:55.048364 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:55.048328 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt"] Oct 30 22:57:55.100928 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:55.100891 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99a8e69-q5jvt"] Oct 30 22:57:55.320570 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:57:55.320534 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:57:57.057370 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:57:57.057338 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edf5288d-d3d4-4613-96cc-f12f166f4112" path="/var/lib/kubelet/pods/edf5288d-d3d4-4613-96cc-f12f166f4112/volumes" Oct 30 22:58:25.321049 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:58:25.321007 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:58:55.320100 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:58:55.320016 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:59:05.709357 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:05.709322 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7"] Oct 30 22:59:05.713001 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:05.712984 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7" Oct 30 22:59:05.771408 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:05.771384 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7"] Oct 30 22:59:05.816837 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:05.816811 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:59:05.817002 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:05.816854 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:59:05.817002 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:05.816887 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:59:05.817002 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:05.816933 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvx84\"" Oct 30 22:59:05.839580 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:05.839558 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7\" (UID: \"ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7" Oct 30 22:59:05.839695 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:05.839588 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh8s6\" (UniqueName: \"kubernetes.io/projected/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-kube-api-access-zh8s6\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7\" (UID: \"ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7" Oct 30 22:59:05.839769 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:05.839751 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7\" (UID: \"ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7" Oct 30 22:59:05.940521 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:05.940500 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7\" (UID: \"ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7" Oct 30 22:59:05.940653 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:05.940553 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7\" (UID: \"ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7" Oct 30 22:59:05.940653 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:05.940582 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8s6\" (UniqueName: \"kubernetes.io/projected/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-kube-api-access-zh8s6\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7\" (UID: \"ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7" Oct 30 22:59:05.941276 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:05.941254 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7\" (UID: \"ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7" Oct 30 22:59:05.943054 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:05.943034 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7\" (UID: \"ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7" Oct 30 22:59:06.001064 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:06.001015 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh8s6\" (UniqueName: \"kubernetes.io/projected/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-kube-api-access-zh8s6\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7\" (UID: \"ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7" Oct 30 22:59:06.025863 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:06.025843 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7" Oct 30 22:59:06.402923 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:06.402894 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7"] Oct 30 22:59:06.404082 ip-10-0-131-193 kubenswrapper[4335]: W1030 22:59:06.404059 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab85f30f_7d21_4c4e_8fdc_1bf2dd1e4178.slice/crio-c891c7af0510b80a5c420452dc38ea20aa84d333784bb5102b8ece292fabf89d WatchSource:0}: Error finding container c891c7af0510b80a5c420452dc38ea20aa84d333784bb5102b8ece292fabf89d: Status 404 returned error can't find the container with id c891c7af0510b80a5c420452dc38ea20aa84d333784bb5102b8ece292fabf89d Oct 30 22:59:07.356556 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:07.356525 4335 generic.go:356] "Generic (PLEG): container finished" podID="ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178" containerID="65d8f398be53cdb5b31b92bc911a26b0361879d2d2ac9e770b655c668973ed5b" exitCode=0 Oct 30 22:59:07.356898 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:07.356637 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7" event={"ID":"ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178","Type":"ContainerDied","Data":"65d8f398be53cdb5b31b92bc911a26b0361879d2d2ac9e770b655c668973ed5b"} Oct 30 22:59:07.356898 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:07.356703 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7" event={"ID":"ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178","Type":"ContainerStarted","Data":"c891c7af0510b80a5c420452dc38ea20aa84d333784bb5102b8ece292fabf89d"} Oct 30 22:59:08.506765 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:08.506739 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7" Oct 30 22:59:08.560913 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:08.560891 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-kube-api-access-zh8s6\" (UniqueName: \"kubernetes.io/projected/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-kube-api-access-zh8s6\") pod \"ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178\" (UID: \"ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178\") " Oct 30 22:59:08.561048 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:08.560939 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-maas-api-xks-ca\") pod \"ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178\" (UID: \"ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178\") " Oct 30 22:59:08.561048 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:08.561004 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-maas-api-openshift-ca\") pod \"ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178\" (UID: \"ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178\") " Oct 30 22:59:08.561496 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:08.561471 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-maas-api-openshift-ca" pod "ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178" (UID: "ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:59:08.564694 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:08.564650 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-maas-api-xks-ca" pod "ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178" (UID: "ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:59:08.565524 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:08.565501 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-kube-api-access-zh8s6" pod "ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178" (UID: "ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178"). InnerVolumeSpecName "kube-api-access-zh8s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:59:08.661762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:08.661713 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:59:08.661762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:08.661734 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh8s6\" (UniqueName: \"kubernetes.io/projected/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-kube-api-access-zh8s6\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:59:08.661762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:08.661745 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:59:09.370128 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:09.370066 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7" Oct 30 22:59:09.370266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:09.370113 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7" event={"ID":"ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178","Type":"ContainerDied","Data":"c891c7af0510b80a5c420452dc38ea20aa84d333784bb5102b8ece292fabf89d"} Oct 30 22:59:09.370266 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:09.370169 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c891c7af0510b80a5c420452dc38ea20aa84d333784bb5102b8ece292fabf89d" Oct 30 22:59:16.083434 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:16.083394 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7"] Oct 30 22:59:16.193643 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:16.193580 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp"] Oct 30 22:59:21.084283 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.084244 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7" podUID="1635fc97-dc4b-4e6a-a842-39f39135d803" containerName="payload-processing" containerID="cri-o://2e90e614d5e09aa5f59602b891a59eadd56be4705e98b33ecbf4096fb4026563" gracePeriod=25 Oct 30 22:59:21.194233 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.194195 4335 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp" podUID="a86627d3-cce0-41b1-9e88-12bffcdd0a3c" containerName="payload-pre-processing" containerID="cri-o://0e80c953ba2e7f5ca423631a778e62524ba2935185613d4b4b929cec45d27991" gracePeriod=25 Oct 30 22:59:21.341514 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.340503 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7" Oct 30 22:59:21.381143 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.381119 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1635fc97-dc4b-4e6a-a842-39f39135d803-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1635fc97-dc4b-4e6a-a842-39f39135d803-plugins-config-volume\") pod \"1635fc97-dc4b-4e6a-a842-39f39135d803\" (UID: \"1635fc97-dc4b-4e6a-a842-39f39135d803\") " Oct 30 22:59:21.381289 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.381176 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1635fc97-dc4b-4e6a-a842-39f39135d803-kube-api-access-r5xdf\" (UniqueName: \"kubernetes.io/projected/1635fc97-dc4b-4e6a-a842-39f39135d803-kube-api-access-r5xdf\") pod \"1635fc97-dc4b-4e6a-a842-39f39135d803\" (UID: \"1635fc97-dc4b-4e6a-a842-39f39135d803\") " Oct 30 22:59:21.381510 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.381485 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1635fc97-dc4b-4e6a-a842-39f39135d803-plugins-config-volume" pod "1635fc97-dc4b-4e6a-a842-39f39135d803" (UID: "1635fc97-dc4b-4e6a-a842-39f39135d803"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:59:21.383824 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.383800 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1635fc97-dc4b-4e6a-a842-39f39135d803-kube-api-access-r5xdf" pod "1635fc97-dc4b-4e6a-a842-39f39135d803" (UID: "1635fc97-dc4b-4e6a-a842-39f39135d803"). InnerVolumeSpecName "kube-api-access-r5xdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:59:21.438572 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.438545 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp" Oct 30 22:59:21.443824 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.443800 4335 generic.go:356] "Generic (PLEG): container finished" podID="a86627d3-cce0-41b1-9e88-12bffcdd0a3c" containerID="0e80c953ba2e7f5ca423631a778e62524ba2935185613d4b4b929cec45d27991" exitCode=0 Oct 30 22:59:21.443927 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.443844 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp" Oct 30 22:59:21.443927 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.443880 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp" event={"ID":"a86627d3-cce0-41b1-9e88-12bffcdd0a3c","Type":"ContainerDied","Data":"0e80c953ba2e7f5ca423631a778e62524ba2935185613d4b4b929cec45d27991"} Oct 30 22:59:21.444015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.443933 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp" event={"ID":"a86627d3-cce0-41b1-9e88-12bffcdd0a3c","Type":"ContainerDied","Data":"31bd2229a142b9e0769711993ba750452679dd9ef5524b91f395fb305b632f1d"} Oct 30 22:59:21.444015 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.443958 4335 scope.go:122] "RemoveContainer" containerID="0e80c953ba2e7f5ca423631a778e62524ba2935185613d4b4b929cec45d27991" Oct 30 22:59:21.445342 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.445319 4335 generic.go:356] "Generic (PLEG): container finished" podID="1635fc97-dc4b-4e6a-a842-39f39135d803" containerID="2e90e614d5e09aa5f59602b891a59eadd56be4705e98b33ecbf4096fb4026563" exitCode=0 Oct 30 22:59:21.445453 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.445358 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7" Oct 30 22:59:21.445453 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.445378 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7" event={"ID":"1635fc97-dc4b-4e6a-a842-39f39135d803","Type":"ContainerDied","Data":"2e90e614d5e09aa5f59602b891a59eadd56be4705e98b33ecbf4096fb4026563"} Oct 30 22:59:21.445574 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.445559 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7" event={"ID":"1635fc97-dc4b-4e6a-a842-39f39135d803","Type":"ContainerDied","Data":"8c1376812603e77b8c3627301c783964521368cf22411cd4741204e63eca228a"} Oct 30 22:59:21.453879 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.453860 4335 scope.go:122] "RemoveContainer" containerID="0e80c953ba2e7f5ca423631a778e62524ba2935185613d4b4b929cec45d27991" Oct 30 22:59:21.454131 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:59:21.454115 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e80c953ba2e7f5ca423631a778e62524ba2935185613d4b4b929cec45d27991\": container with ID starting with 0e80c953ba2e7f5ca423631a778e62524ba2935185613d4b4b929cec45d27991 not found: ID does not exist" containerID="0e80c953ba2e7f5ca423631a778e62524ba2935185613d4b4b929cec45d27991" Oct 30 22:59:21.454184 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.454139 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e80c953ba2e7f5ca423631a778e62524ba2935185613d4b4b929cec45d27991"} err="failed to get container status \"0e80c953ba2e7f5ca423631a778e62524ba2935185613d4b4b929cec45d27991\": rpc error: code = NotFound desc = could not find container \"0e80c953ba2e7f5ca423631a778e62524ba2935185613d4b4b929cec45d27991\": container with ID starting with 0e80c953ba2e7f5ca423631a778e62524ba2935185613d4b4b929cec45d27991 not found: ID does not exist" Oct 30 22:59:21.454184 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.454154 4335 scope.go:122] "RemoveContainer" containerID="2e90e614d5e09aa5f59602b891a59eadd56be4705e98b33ecbf4096fb4026563" Oct 30 22:59:21.463441 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.463416 4335 scope.go:122] "RemoveContainer" containerID="2e90e614d5e09aa5f59602b891a59eadd56be4705e98b33ecbf4096fb4026563" Oct 30 22:59:21.463831 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:59:21.463778 4335 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e90e614d5e09aa5f59602b891a59eadd56be4705e98b33ecbf4096fb4026563\": container with ID starting with 2e90e614d5e09aa5f59602b891a59eadd56be4705e98b33ecbf4096fb4026563 not found: ID does not exist" containerID="2e90e614d5e09aa5f59602b891a59eadd56be4705e98b33ecbf4096fb4026563" Oct 30 22:59:21.463831 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.463807 4335 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e90e614d5e09aa5f59602b891a59eadd56be4705e98b33ecbf4096fb4026563"} err="failed to get container status \"2e90e614d5e09aa5f59602b891a59eadd56be4705e98b33ecbf4096fb4026563\": rpc error: code = NotFound desc = could not find container \"2e90e614d5e09aa5f59602b891a59eadd56be4705e98b33ecbf4096fb4026563\": container with ID starting with 2e90e614d5e09aa5f59602b891a59eadd56be4705e98b33ecbf4096fb4026563 not found: ID does not exist" Oct 30 22:59:21.481848 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.481827 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a86627d3-cce0-41b1-9e88-12bffcdd0a3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a86627d3-cce0-41b1-9e88-12bffcdd0a3c-plugins-config-volume\") pod \"a86627d3-cce0-41b1-9e88-12bffcdd0a3c\" (UID: \"a86627d3-cce0-41b1-9e88-12bffcdd0a3c\") " Oct 30 22:59:21.481959 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.481881 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a86627d3-cce0-41b1-9e88-12bffcdd0a3c-kube-api-access-8hk4p\" (UniqueName: \"kubernetes.io/projected/a86627d3-cce0-41b1-9e88-12bffcdd0a3c-kube-api-access-8hk4p\") pod \"a86627d3-cce0-41b1-9e88-12bffcdd0a3c\" (UID: \"a86627d3-cce0-41b1-9e88-12bffcdd0a3c\") " Oct 30 22:59:21.482196 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.482181 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1635fc97-dc4b-4e6a-a842-39f39135d803-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:59:21.482259 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.482199 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5xdf\" (UniqueName: \"kubernetes.io/projected/1635fc97-dc4b-4e6a-a842-39f39135d803-kube-api-access-r5xdf\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:59:21.482259 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.482236 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a86627d3-cce0-41b1-9e88-12bffcdd0a3c-plugins-config-volume" pod "a86627d3-cce0-41b1-9e88-12bffcdd0a3c" (UID: "a86627d3-cce0-41b1-9e88-12bffcdd0a3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:59:21.484058 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.484035 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a86627d3-cce0-41b1-9e88-12bffcdd0a3c-kube-api-access-8hk4p" pod "a86627d3-cce0-41b1-9e88-12bffcdd0a3c" (UID: "a86627d3-cce0-41b1-9e88-12bffcdd0a3c"). InnerVolumeSpecName "kube-api-access-8hk4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:59:21.571517 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.571496 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7"] Oct 30 22:59:21.576128 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.576108 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8f0848-bc5b4cdd7-5fgl7"] Oct 30 22:59:21.583578 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.583558 4335 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a86627d3-cce0-41b1-9e88-12bffcdd0a3c-plugins-config-volume\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:59:21.583667 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.583580 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hk4p\" (UniqueName: \"kubernetes.io/projected/a86627d3-cce0-41b1-9e88-12bffcdd0a3c-kube-api-access-8hk4p\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:59:21.864338 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.864314 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp"] Oct 30 22:59:21.870875 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:21.870855 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8f0848-6f4db745d7-4q8sp"] Oct 30 22:59:23.055762 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:23.055731 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1635fc97-dc4b-4e6a-a842-39f39135d803" path="/var/lib/kubelet/pods/1635fc97-dc4b-4e6a-a842-39f39135d803/volumes" Oct 30 22:59:23.056151 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:23.056068 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a86627d3-cce0-41b1-9e88-12bffcdd0a3c" path="/var/lib/kubelet/pods/a86627d3-cce0-41b1-9e88-12bffcdd0a3c/volumes" Oct 30 22:59:25.320422 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:59:25.320387 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:59:29.031424 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:29.031393 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:59:29.061024 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:29.060997 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9bdx_2a6e38bc-3e75-4392-90b2-a0472b37a443/extract/0.log" Oct 30 22:59:31.413855 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:31.413825 4335 scope.go:122] "RemoveContainer" containerID="4774c5ba9cfc8e40d2a94793073ee971fbb530a8554654f0233613b96fce4261" Oct 30 22:59:31.423591 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:31.423574 4335 scope.go:122] "RemoveContainer" containerID="df1b863bb74a9da9071484bd807feef7c98f2d684f044279de71b3e4ca43a09a" Oct 30 22:59:31.458048 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:31.458026 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7"] Oct 30 22:59:31.507137 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:31.507114 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8f0848-z9bg7"] Oct 30 22:59:33.056551 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:33.056520 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178" path="/var/lib/kubelet/pods/ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178/volumes" Oct 30 22:59:34.455218 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:34.455173 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:59:44.738552 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:44.738516 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:59:54.747545 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:54.747510 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 22:59:55.333359 ip-10-0-131-193 kubenswrapper[4335]: E1030 22:59:55.333316 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 22:59:59.048448 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:59.048418 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j"] Oct 30 22:59:59.097938 ip-10-0-131-193 kubenswrapper[4335]: I1030 22:59:59.097906 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-x7z9j"] Oct 30 23:00:00.180892 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.180860 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd"] Oct 30 23:00:00.181478 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.181457 4335 cpu_manager.go:414] "containerMap: removing container" podUID="1635fc97-dc4b-4e6a-a842-39f39135d803" containerName="payload-processing" Oct 30 23:00:00.181568 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.181480 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1635fc97-dc4b-4e6a-a842-39f39135d803" containerName="payload-processing" Oct 30 23:00:00.181568 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.181502 4335 cpu_manager.go:414] "containerMap: removing container" podUID="a86627d3-cce0-41b1-9e88-12bffcdd0a3c" containerName="payload-pre-processing" Oct 30 23:00:00.181568 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.181509 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a86627d3-cce0-41b1-9e88-12bffcdd0a3c" containerName="payload-pre-processing" Oct 30 23:00:00.181568 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.181536 4335 cpu_manager.go:414] "containerMap: removing container" podUID="ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178" containerName="revoke-keys" Oct 30 23:00:00.181568 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.181543 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178" containerName="revoke-keys" Oct 30 23:00:00.181887 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.181671 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="1635fc97-dc4b-4e6a-a842-39f39135d803" containerName="payload-processing" Oct 30 23:00:00.181887 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.181688 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="a86627d3-cce0-41b1-9e88-12bffcdd0a3c" containerName="payload-pre-processing" Oct 30 23:00:00.181887 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.181702 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab85f30f-7d21-4c4e-8fdc-1bf2dd1e4178" containerName="revoke-keys" Oct 30 23:00:00.185095 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.185074 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd" Oct 30 23:00:00.240744 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.240720 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d203b0-eae9-466a-b5b3-632daa69c213-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890020-nmtfd\" (UID: \"f3d203b0-eae9-466a-b5b3-632daa69c213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd" Oct 30 23:00:00.240875 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.240857 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7nm\" (UniqueName: \"kubernetes.io/projected/f3d203b0-eae9-466a-b5b3-632daa69c213-kube-api-access-st7nm\") pod \"maas-api-key-cleanup-29890020-nmtfd\" (UID: \"f3d203b0-eae9-466a-b5b3-632daa69c213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd" Oct 30 23:00:00.287659 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.287631 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:00:00.287659 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.287643 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:00:00.287792 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.287709 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvx84\"" Oct 30 23:00:00.290467 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.290448 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd"] Oct 30 23:00:00.341981 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.341959 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7nm\" (UniqueName: \"kubernetes.io/projected/f3d203b0-eae9-466a-b5b3-632daa69c213-kube-api-access-st7nm\") pod \"maas-api-key-cleanup-29890020-nmtfd\" (UID: \"f3d203b0-eae9-466a-b5b3-632daa69c213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd" Oct 30 23:00:00.342081 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.342019 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d203b0-eae9-466a-b5b3-632daa69c213-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890020-nmtfd\" (UID: \"f3d203b0-eae9-466a-b5b3-632daa69c213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd" Oct 30 23:00:00.342586 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.342569 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d203b0-eae9-466a-b5b3-632daa69c213-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890020-nmtfd\" (UID: \"f3d203b0-eae9-466a-b5b3-632daa69c213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd" Oct 30 23:00:00.403289 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.403266 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7nm\" (UniqueName: \"kubernetes.io/projected/f3d203b0-eae9-466a-b5b3-632daa69c213-kube-api-access-st7nm\") pod \"maas-api-key-cleanup-29890020-nmtfd\" (UID: \"f3d203b0-eae9-466a-b5b3-632daa69c213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd" Oct 30 23:00:00.497653 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.497580 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd" Oct 30 23:00:00.875368 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:00.875342 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd"] Oct 30 23:00:00.876882 ip-10-0-131-193 kubenswrapper[4335]: W1030 23:00:00.876846 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d203b0_eae9_466a_b5b3_632daa69c213.slice/crio-5db238098878b9cbcd511107864ecd08cece54e1a0e33d1f09c81e7eb87b2907 WatchSource:0}: Error finding container 5db238098878b9cbcd511107864ecd08cece54e1a0e33d1f09c81e7eb87b2907: Status 404 returned error can't find the container with id 5db238098878b9cbcd511107864ecd08cece54e1a0e33d1f09c81e7eb87b2907 Oct 30 23:00:01.056491 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:01.056453 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="960b0f4e-0d48-4009-b141-19579f9963d0" path="/var/lib/kubelet/pods/960b0f4e-0d48-4009-b141-19579f9963d0/volumes" Oct 30 23:00:01.781781 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:01.781749 4335 generic.go:356] "Generic (PLEG): container finished" podID="f3d203b0-eae9-466a-b5b3-632daa69c213" containerID="80038907547a1af51b7f640afbeec27352660d66410c744037d7e6c692701762" exitCode=60 Oct 30 23:00:01.782119 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:01.781831 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd" event={"ID":"f3d203b0-eae9-466a-b5b3-632daa69c213","Type":"ContainerDied","Data":"80038907547a1af51b7f640afbeec27352660d66410c744037d7e6c692701762"} Oct 30 23:00:01.782119 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:01.781883 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd" event={"ID":"f3d203b0-eae9-466a-b5b3-632daa69c213","Type":"ContainerStarted","Data":"5db238098878b9cbcd511107864ecd08cece54e1a0e33d1f09c81e7eb87b2907"} Oct 30 23:00:01.782119 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:01.782026 4335 scope.go:122] "RemoveContainer" containerID="80038907547a1af51b7f640afbeec27352660d66410c744037d7e6c692701762" Oct 30 23:00:02.790256 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:02.790176 4335 generic.go:356] "Generic (PLEG): container finished" podID="f3d203b0-eae9-466a-b5b3-632daa69c213" containerID="fea03d6134f63f43b17ac1c4fe72890532026266de878c261a870af902aa46b3" exitCode=60 Oct 30 23:00:02.790701 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:02.790251 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd" event={"ID":"f3d203b0-eae9-466a-b5b3-632daa69c213","Type":"ContainerDied","Data":"fea03d6134f63f43b17ac1c4fe72890532026266de878c261a870af902aa46b3"} Oct 30 23:00:02.790701 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:02.790304 4335 scope.go:122] "RemoveContainer" containerID="80038907547a1af51b7f640afbeec27352660d66410c744037d7e6c692701762" Oct 30 23:00:02.790701 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:02.790567 4335 scope.go:122] "RemoveContainer" containerID="fea03d6134f63f43b17ac1c4fe72890532026266de878c261a870af902aa46b3" Oct 30 23:00:02.790853 ip-10-0-131-193 kubenswrapper[4335]: E1030 23:00:02.790781 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890020-nmtfd_odh-ai-gateway-infra(f3d203b0-eae9-466a-b5b3-632daa69c213)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd" podUID="f3d203b0-eae9-466a-b5b3-632daa69c213" Oct 30 23:00:03.797911 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:03.797882 4335 scope.go:122] "RemoveContainer" containerID="fea03d6134f63f43b17ac1c4fe72890532026266de878c261a870af902aa46b3" Oct 30 23:00:03.798404 ip-10-0-131-193 kubenswrapper[4335]: E1030 23:00:03.798140 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890020-nmtfd_odh-ai-gateway-infra(f3d203b0-eae9-466a-b5b3-632daa69c213)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd" podUID="f3d203b0-eae9-466a-b5b3-632daa69c213" Oct 30 23:00:05.547823 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:05.547793 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp"] Oct 30 23:00:05.552558 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:05.552539 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp" Oct 30 23:00:05.587422 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:05.587393 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-8dxlp\" (UID: \"abc66f6e-fa6c-41be-9221-4cbbcfa6db58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp" Oct 30 23:00:05.587514 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:05.587443 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-757g7\" (UniqueName: \"kubernetes.io/projected/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-kube-api-access-757g7\") pod \"maas-api-revoke-sub-f03629db-8dxlp\" (UID: \"abc66f6e-fa6c-41be-9221-4cbbcfa6db58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp" Oct 30 23:00:05.587599 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:05.587576 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-8dxlp\" (UID: \"abc66f6e-fa6c-41be-9221-4cbbcfa6db58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp" Oct 30 23:00:05.605888 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:05.605859 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:00:05.658163 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:05.658139 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp"] Oct 30 23:00:05.688369 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:05.688343 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-8dxlp\" (UID: \"abc66f6e-fa6c-41be-9221-4cbbcfa6db58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp" Oct 30 23:00:05.688481 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:05.688372 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-8dxlp\" (UID: \"abc66f6e-fa6c-41be-9221-4cbbcfa6db58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp" Oct 30 23:00:05.688481 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:05.688409 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-757g7\" (UniqueName: \"kubernetes.io/projected/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-kube-api-access-757g7\") pod \"maas-api-revoke-sub-f03629db-8dxlp\" (UID: \"abc66f6e-fa6c-41be-9221-4cbbcfa6db58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp" Oct 30 23:00:05.688995 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:05.688972 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-8dxlp\" (UID: \"abc66f6e-fa6c-41be-9221-4cbbcfa6db58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp" Oct 30 23:00:05.690851 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:05.690830 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-8dxlp\" (UID: \"abc66f6e-fa6c-41be-9221-4cbbcfa6db58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp" Oct 30 23:00:05.768182 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:05.768157 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-757g7\" (UniqueName: \"kubernetes.io/projected/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-kube-api-access-757g7\") pod \"maas-api-revoke-sub-f03629db-8dxlp\" (UID: \"abc66f6e-fa6c-41be-9221-4cbbcfa6db58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp" Oct 30 23:00:05.865299 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:05.865279 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp" Oct 30 23:00:06.241510 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:06.241486 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp"] Oct 30 23:00:06.242380 ip-10-0-131-193 kubenswrapper[4335]: W1030 23:00:06.242358 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc66f6e_fa6c_41be_9221_4cbbcfa6db58.slice/crio-63dc85b15186e4fbd8fe1217065ae52629298273dea5f3849ac52319a55fcbc7 WatchSource:0}: Error finding container 63dc85b15186e4fbd8fe1217065ae52629298273dea5f3849ac52319a55fcbc7: Status 404 returned error can't find the container with id 63dc85b15186e4fbd8fe1217065ae52629298273dea5f3849ac52319a55fcbc7 Oct 30 23:00:06.821277 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:06.821246 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp" event={"ID":"abc66f6e-fa6c-41be-9221-4cbbcfa6db58","Type":"ContainerStarted","Data":"8c5fcdf8c783e82a18807a250a6febc60c987f085e5a4d85f58a676853fe1906"} Oct 30 23:00:06.821605 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:06.821306 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp" event={"ID":"abc66f6e-fa6c-41be-9221-4cbbcfa6db58","Type":"ContainerStarted","Data":"63dc85b15186e4fbd8fe1217065ae52629298273dea5f3849ac52319a55fcbc7"} Oct 30 23:00:06.938601 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:06.938557 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp" podStartSLOduration=1.93854496 podStartE2EDuration="1.93854496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:06.935817307 +0000 UTC m=+4540.036249055" watchObservedRunningTime="2026-10-30 23:00:06.93854496 +0000 UTC m=+4540.038976713" Oct 30 23:00:07.828761 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:07.828727 4335 generic.go:356] "Generic (PLEG): container finished" podID="abc66f6e-fa6c-41be-9221-4cbbcfa6db58" containerID="8c5fcdf8c783e82a18807a250a6febc60c987f085e5a4d85f58a676853fe1906" exitCode=0 Oct 30 23:00:07.829171 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:07.828811 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp" event={"ID":"abc66f6e-fa6c-41be-9221-4cbbcfa6db58","Type":"ContainerDied","Data":"8c5fcdf8c783e82a18807a250a6febc60c987f085e5a4d85f58a676853fe1906"} Oct 30 23:00:08.959524 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:08.959495 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp" Oct 30 23:00:09.014857 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:09.014832 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-kube-api-access-757g7\" (UniqueName: \"kubernetes.io/projected/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-kube-api-access-757g7\") pod \"abc66f6e-fa6c-41be-9221-4cbbcfa6db58\" (UID: \"abc66f6e-fa6c-41be-9221-4cbbcfa6db58\") " Oct 30 23:00:09.015012 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:09.014876 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-maas-api-openshift-ca\") pod \"abc66f6e-fa6c-41be-9221-4cbbcfa6db58\" (UID: \"abc66f6e-fa6c-41be-9221-4cbbcfa6db58\") " Oct 30 23:00:09.015092 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:09.015006 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-maas-api-xks-ca\") pod \"abc66f6e-fa6c-41be-9221-4cbbcfa6db58\" (UID: \"abc66f6e-fa6c-41be-9221-4cbbcfa6db58\") " Oct 30 23:00:09.015195 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:09.015173 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-maas-api-openshift-ca" pod "abc66f6e-fa6c-41be-9221-4cbbcfa6db58" (UID: "abc66f6e-fa6c-41be-9221-4cbbcfa6db58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:00:09.015357 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:09.015339 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:09.017060 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:09.017035 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-maas-api-xks-ca" pod "abc66f6e-fa6c-41be-9221-4cbbcfa6db58" (UID: "abc66f6e-fa6c-41be-9221-4cbbcfa6db58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:00:09.017209 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:09.017186 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-kube-api-access-757g7" pod "abc66f6e-fa6c-41be-9221-4cbbcfa6db58" (UID: "abc66f6e-fa6c-41be-9221-4cbbcfa6db58"). InnerVolumeSpecName "kube-api-access-757g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:00:09.116150 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:09.116097 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-757g7\" (UniqueName: \"kubernetes.io/projected/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-kube-api-access-757g7\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:09.116150 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:09.116119 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc66f6e-fa6c-41be-9221-4cbbcfa6db58-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:09.841389 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:09.841361 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp" Oct 30 23:00:09.841570 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:09.841397 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8dxlp" event={"ID":"abc66f6e-fa6c-41be-9221-4cbbcfa6db58","Type":"ContainerDied","Data":"63dc85b15186e4fbd8fe1217065ae52629298273dea5f3849ac52319a55fcbc7"} Oct 30 23:00:09.841570 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:09.841441 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63dc85b15186e4fbd8fe1217065ae52629298273dea5f3849ac52319a55fcbc7" Oct 30 23:00:14.050730 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:14.050702 4335 scope.go:122] "RemoveContainer" containerID="fea03d6134f63f43b17ac1c4fe72890532026266de878c261a870af902aa46b3" Oct 30 23:00:14.879093 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:14.879059 4335 generic.go:356] "Generic (PLEG): container finished" podID="f3d203b0-eae9-466a-b5b3-632daa69c213" containerID="c49f88129015b9a5ee84c7626048a84d45cf41fd5c23eddd952e664b332761cc" exitCode=60 Oct 30 23:00:14.879249 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:14.879131 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd" event={"ID":"f3d203b0-eae9-466a-b5b3-632daa69c213","Type":"ContainerDied","Data":"c49f88129015b9a5ee84c7626048a84d45cf41fd5c23eddd952e664b332761cc"} Oct 30 23:00:14.879249 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:14.879187 4335 scope.go:122] "RemoveContainer" containerID="fea03d6134f63f43b17ac1c4fe72890532026266de878c261a870af902aa46b3" Oct 30 23:00:14.879518 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:14.879499 4335 scope.go:122] "RemoveContainer" containerID="c49f88129015b9a5ee84c7626048a84d45cf41fd5c23eddd952e664b332761cc" Oct 30 23:00:14.879763 ip-10-0-131-193 kubenswrapper[4335]: E1030 23:00:14.879740 4335 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890020-nmtfd_odh-ai-gateway-infra(f3d203b0-eae9-466a-b5b3-632daa69c213)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd" podUID="f3d203b0-eae9-466a-b5b3-632daa69c213" Oct 30 23:00:16.005587 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:16.005557 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd"] Oct 30 23:00:16.149082 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:16.149059 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd" Oct 30 23:00:16.274726 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:16.274674 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3d203b0-eae9-466a-b5b3-632daa69c213-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d203b0-eae9-466a-b5b3-632daa69c213-maas-api-service-ca\") pod \"f3d203b0-eae9-466a-b5b3-632daa69c213\" (UID: \"f3d203b0-eae9-466a-b5b3-632daa69c213\") " Oct 30 23:00:16.274869 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:16.274742 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3d203b0-eae9-466a-b5b3-632daa69c213-kube-api-access-st7nm\" (UniqueName: \"kubernetes.io/projected/f3d203b0-eae9-466a-b5b3-632daa69c213-kube-api-access-st7nm\") pod \"f3d203b0-eae9-466a-b5b3-632daa69c213\" (UID: \"f3d203b0-eae9-466a-b5b3-632daa69c213\") " Oct 30 23:00:16.275082 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:16.275057 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3d203b0-eae9-466a-b5b3-632daa69c213-maas-api-service-ca" pod "f3d203b0-eae9-466a-b5b3-632daa69c213" (UID: "f3d203b0-eae9-466a-b5b3-632daa69c213"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:00:16.276847 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:16.276828 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d203b0-eae9-466a-b5b3-632daa69c213-kube-api-access-st7nm" pod "f3d203b0-eae9-466a-b5b3-632daa69c213" (UID: "f3d203b0-eae9-466a-b5b3-632daa69c213"). InnerVolumeSpecName "kube-api-access-st7nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:00:16.375729 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:16.375699 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d203b0-eae9-466a-b5b3-632daa69c213-maas-api-service-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:16.375729 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:16.375728 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st7nm\" (UniqueName: \"kubernetes.io/projected/f3d203b0-eae9-466a-b5b3-632daa69c213-kube-api-access-st7nm\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:16.894895 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:16.894867 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd" Oct 30 23:00:16.895074 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:16.894873 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd" event={"ID":"f3d203b0-eae9-466a-b5b3-632daa69c213","Type":"ContainerDied","Data":"5db238098878b9cbcd511107864ecd08cece54e1a0e33d1f09c81e7eb87b2907"} Oct 30 23:00:16.895074 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:16.894996 4335 scope.go:122] "RemoveContainer" containerID="c49f88129015b9a5ee84c7626048a84d45cf41fd5c23eddd952e664b332761cc" Oct 30 23:00:17.058851 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:17.058827 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd"] Oct 30 23:00:17.110019 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:17.109989 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-nmtfd"] Oct 30 23:00:18.431915 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:18.431882 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 23:00:19.056494 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:19.056464 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3d203b0-eae9-466a-b5b3-632daa69c213" path="/var/lib/kubelet/pods/f3d203b0-eae9-466a-b5b3-632daa69c213/volumes" Oct 30 23:00:20.049276 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:20.049241 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv"] Oct 30 23:00:20.098389 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:20.098355 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zc7bv"] Oct 30 23:00:21.055437 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:21.055405 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af" path="/var/lib/kubelet/pods/3a49a8c2-6933-4e2f-a2aa-2abb5a7db9af/volumes" Oct 30 23:00:24.711764 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.711713 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9"] Oct 30 23:00:24.712979 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.712953 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f3d203b0-eae9-466a-b5b3-632daa69c213" containerName="cleanup" Oct 30 23:00:24.717647 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.713125 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3d203b0-eae9-466a-b5b3-632daa69c213" containerName="cleanup" Oct 30 23:00:24.719061 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.719044 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f3d203b0-eae9-466a-b5b3-632daa69c213" containerName="cleanup" Oct 30 23:00:24.719214 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.719199 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3d203b0-eae9-466a-b5b3-632daa69c213" containerName="cleanup" Oct 30 23:00:24.719359 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.719346 4335 cpu_manager.go:414] "containerMap: removing container" podUID="abc66f6e-fa6c-41be-9221-4cbbcfa6db58" containerName="revoke-keys" Oct 30 23:00:24.719473 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.719460 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc66f6e-fa6c-41be-9221-4cbbcfa6db58" containerName="revoke-keys" Oct 30 23:00:24.723691 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.723674 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3d203b0-eae9-466a-b5b3-632daa69c213" containerName="cleanup" Oct 30 23:00:24.723854 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.723840 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc66f6e-fa6c-41be-9221-4cbbcfa6db58" containerName="revoke-keys" Oct 30 23:00:24.724000 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.723987 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3d203b0-eae9-466a-b5b3-632daa69c213" containerName="cleanup" Oct 30 23:00:24.738297 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.738277 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9" Oct 30 23:00:24.781750 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.781725 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9"] Oct 30 23:00:24.825038 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.825007 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvx84\"" Oct 30 23:00:24.825385 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.825333 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:00:24.854894 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.854766 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:00:24.854894 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.854850 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:00:24.863411 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.863099 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea1a5c9f-93c6-4407-8624-62627bc59533-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-bkml9\" (UID: \"ea1a5c9f-93c6-4407-8624-62627bc59533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9" Oct 30 23:00:24.863411 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.863149 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scddb\" (UniqueName: \"kubernetes.io/projected/ea1a5c9f-93c6-4407-8624-62627bc59533-kube-api-access-scddb\") pod \"maas-api-revoke-sub-0165d7d2-bkml9\" (UID: \"ea1a5c9f-93c6-4407-8624-62627bc59533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9" Oct 30 23:00:24.863411 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.863228 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea1a5c9f-93c6-4407-8624-62627bc59533-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-bkml9\" (UID: \"ea1a5c9f-93c6-4407-8624-62627bc59533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9" Oct 30 23:00:24.984699 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.982360 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea1a5c9f-93c6-4407-8624-62627bc59533-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-bkml9\" (UID: \"ea1a5c9f-93c6-4407-8624-62627bc59533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9" Oct 30 23:00:24.984699 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.982437 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scddb\" (UniqueName: \"kubernetes.io/projected/ea1a5c9f-93c6-4407-8624-62627bc59533-kube-api-access-scddb\") pod \"maas-api-revoke-sub-0165d7d2-bkml9\" (UID: \"ea1a5c9f-93c6-4407-8624-62627bc59533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9" Oct 30 23:00:24.984699 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.982538 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea1a5c9f-93c6-4407-8624-62627bc59533-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-bkml9\" (UID: \"ea1a5c9f-93c6-4407-8624-62627bc59533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9" Oct 30 23:00:24.984699 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:24.984092 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea1a5c9f-93c6-4407-8624-62627bc59533-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-bkml9\" (UID: \"ea1a5c9f-93c6-4407-8624-62627bc59533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9" Oct 30 23:00:25.086200 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:25.084491 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea1a5c9f-93c6-4407-8624-62627bc59533-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-bkml9\" (UID: \"ea1a5c9f-93c6-4407-8624-62627bc59533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9" Oct 30 23:00:25.116674 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:25.097323 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scddb\" (UniqueName: \"kubernetes.io/projected/ea1a5c9f-93c6-4407-8624-62627bc59533-kube-api-access-scddb\") pod \"maas-api-revoke-sub-0165d7d2-bkml9\" (UID: \"ea1a5c9f-93c6-4407-8624-62627bc59533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9" Oct 30 23:00:25.387837 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:25.387802 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9" Oct 30 23:00:25.531641 ip-10-0-131-193 kubenswrapper[4335]: E1030 23:00:25.513043 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 23:00:26.058214 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:26.057741 4335 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-mjgkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:00:26.058214 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:26.057801 4335 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mjgkw" podUID="4ed2ae60-d2b4-4ce7-b48b-295a7eabe663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:00:26.330688 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:26.329558 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9"] Oct 30 23:00:27.143625 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:27.143579 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9" event={"ID":"ea1a5c9f-93c6-4407-8624-62627bc59533","Type":"ContainerStarted","Data":"0b80ae9fb643a2beffd5bb199afb8145521dd4d9de13036880f7979808c57cf3"} Oct 30 23:00:27.144015 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:27.143642 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9" event={"ID":"ea1a5c9f-93c6-4407-8624-62627bc59533","Type":"ContainerStarted","Data":"ad282cab739420a5dc48d1bfeb1d90538ad3bec1a26804a3aa14c29db83b493a"} Oct 30 23:00:27.309978 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:27.309930 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9" podStartSLOduration=3.309916298 podStartE2EDuration="3.309916298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:27.258348122 +0000 UTC m=+4560.358779853" watchObservedRunningTime="2026-10-30 23:00:27.309916298 +0000 UTC m=+4560.410348030" Oct 30 23:00:28.151097 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:28.151058 4335 generic.go:356] "Generic (PLEG): container finished" podID="ea1a5c9f-93c6-4407-8624-62627bc59533" containerID="0b80ae9fb643a2beffd5bb199afb8145521dd4d9de13036880f7979808c57cf3" exitCode=0 Oct 30 23:00:28.151545 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:28.151147 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9" event={"ID":"ea1a5c9f-93c6-4407-8624-62627bc59533","Type":"ContainerDied","Data":"0b80ae9fb643a2beffd5bb199afb8145521dd4d9de13036880f7979808c57cf3"} Oct 30 23:00:29.293898 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:29.293872 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9" Oct 30 23:00:29.305995 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:29.305973 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea1a5c9f-93c6-4407-8624-62627bc59533-kube-api-access-scddb\" (UniqueName: \"kubernetes.io/projected/ea1a5c9f-93c6-4407-8624-62627bc59533-kube-api-access-scddb\") pod \"ea1a5c9f-93c6-4407-8624-62627bc59533\" (UID: \"ea1a5c9f-93c6-4407-8624-62627bc59533\") " Oct 30 23:00:29.306097 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:29.306013 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea1a5c9f-93c6-4407-8624-62627bc59533-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea1a5c9f-93c6-4407-8624-62627bc59533-maas-api-openshift-ca\") pod \"ea1a5c9f-93c6-4407-8624-62627bc59533\" (UID: \"ea1a5c9f-93c6-4407-8624-62627bc59533\") " Oct 30 23:00:29.306152 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:29.306134 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea1a5c9f-93c6-4407-8624-62627bc59533-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea1a5c9f-93c6-4407-8624-62627bc59533-maas-api-xks-ca\") pod \"ea1a5c9f-93c6-4407-8624-62627bc59533\" (UID: \"ea1a5c9f-93c6-4407-8624-62627bc59533\") " Oct 30 23:00:29.306346 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:29.306326 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea1a5c9f-93c6-4407-8624-62627bc59533-maas-api-openshift-ca" pod "ea1a5c9f-93c6-4407-8624-62627bc59533" (UID: "ea1a5c9f-93c6-4407-8624-62627bc59533"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:00:29.308667 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:29.308637 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea1a5c9f-93c6-4407-8624-62627bc59533-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:29.316924 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:29.309118 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea1a5c9f-93c6-4407-8624-62627bc59533-kube-api-access-scddb" pod "ea1a5c9f-93c6-4407-8624-62627bc59533" (UID: "ea1a5c9f-93c6-4407-8624-62627bc59533"). InnerVolumeSpecName "kube-api-access-scddb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:00:29.316924 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:29.309774 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea1a5c9f-93c6-4407-8624-62627bc59533-maas-api-xks-ca" pod "ea1a5c9f-93c6-4407-8624-62627bc59533" (UID: "ea1a5c9f-93c6-4407-8624-62627bc59533"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:00:29.409408 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:29.409350 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scddb\" (UniqueName: \"kubernetes.io/projected/ea1a5c9f-93c6-4407-8624-62627bc59533-kube-api-access-scddb\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:29.409408 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:29.409385 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea1a5c9f-93c6-4407-8624-62627bc59533-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:30.164219 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:30.164192 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9" Oct 30 23:00:30.164399 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:30.164234 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bkml9" event={"ID":"ea1a5c9f-93c6-4407-8624-62627bc59533","Type":"ContainerDied","Data":"ad282cab739420a5dc48d1bfeb1d90538ad3bec1a26804a3aa14c29db83b493a"} Oct 30 23:00:30.164399 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:30.164287 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad282cab739420a5dc48d1bfeb1d90538ad3bec1a26804a3aa14c29db83b493a" Oct 30 23:00:36.507431 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.507401 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw"] Oct 30 23:00:36.507852 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.507840 4335 cpu_manager.go:414] "containerMap: removing container" podUID="ea1a5c9f-93c6-4407-8624-62627bc59533" containerName="revoke-keys" Oct 30 23:00:36.507904 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.507854 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea1a5c9f-93c6-4407-8624-62627bc59533" containerName="revoke-keys" Oct 30 23:00:36.507904 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.507884 4335 cpu_manager.go:414] "containerMap: removing container" podUID="f3d203b0-eae9-466a-b5b3-632daa69c213" containerName="cleanup" Oct 30 23:00:36.507904 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.507889 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3d203b0-eae9-466a-b5b3-632daa69c213" containerName="cleanup" Oct 30 23:00:36.508014 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.507971 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3d203b0-eae9-466a-b5b3-632daa69c213" containerName="cleanup" Oct 30 23:00:36.508014 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.507980 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea1a5c9f-93c6-4407-8624-62627bc59533" containerName="revoke-keys" Oct 30 23:00:36.512574 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.512558 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw" Oct 30 23:00:36.564912 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.564886 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:00:36.565063 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.564963 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:00:36.569460 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.569435 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltzrw\" (UniqueName: \"kubernetes.io/projected/c0e0cc0e-8e59-4023-a501-15caeeaeb538-kube-api-access-ltzrw\") pod \"maas-api-revoke-sub-6b19338f-btljw\" (UID: \"c0e0cc0e-8e59-4023-a501-15caeeaeb538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw" Oct 30 23:00:36.569580 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.569512 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0e0cc0e-8e59-4023-a501-15caeeaeb538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-btljw\" (UID: \"c0e0cc0e-8e59-4023-a501-15caeeaeb538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw" Oct 30 23:00:36.569678 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.569655 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0e0cc0e-8e59-4023-a501-15caeeaeb538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-btljw\" (UID: \"c0e0cc0e-8e59-4023-a501-15caeeaeb538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw" Oct 30 23:00:36.614274 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.614247 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:00:36.614411 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.614392 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvx84\"" Oct 30 23:00:36.617471 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.617449 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw"] Oct 30 23:00:36.670484 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.670456 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0e0cc0e-8e59-4023-a501-15caeeaeb538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-btljw\" (UID: \"c0e0cc0e-8e59-4023-a501-15caeeaeb538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw" Oct 30 23:00:36.670593 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.670508 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0e0cc0e-8e59-4023-a501-15caeeaeb538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-btljw\" (UID: \"c0e0cc0e-8e59-4023-a501-15caeeaeb538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw" Oct 30 23:00:36.670700 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.670591 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltzrw\" (UniqueName: \"kubernetes.io/projected/c0e0cc0e-8e59-4023-a501-15caeeaeb538-kube-api-access-ltzrw\") pod \"maas-api-revoke-sub-6b19338f-btljw\" (UID: \"c0e0cc0e-8e59-4023-a501-15caeeaeb538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw" Oct 30 23:00:36.671121 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.671104 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0e0cc0e-8e59-4023-a501-15caeeaeb538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-btljw\" (UID: \"c0e0cc0e-8e59-4023-a501-15caeeaeb538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw" Oct 30 23:00:36.672945 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.672926 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0e0cc0e-8e59-4023-a501-15caeeaeb538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-btljw\" (UID: \"c0e0cc0e-8e59-4023-a501-15caeeaeb538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw" Oct 30 23:00:36.731718 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.731695 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltzrw\" (UniqueName: \"kubernetes.io/projected/c0e0cc0e-8e59-4023-a501-15caeeaeb538-kube-api-access-ltzrw\") pod \"maas-api-revoke-sub-6b19338f-btljw\" (UID: \"c0e0cc0e-8e59-4023-a501-15caeeaeb538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw" Oct 30 23:00:36.825951 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:36.825930 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw" Oct 30 23:00:37.204342 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:37.204310 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw"] Oct 30 23:00:37.205725 ip-10-0-131-193 kubenswrapper[4335]: W1030 23:00:37.205685 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e0cc0e_8e59_4023_a501_15caeeaeb538.slice/crio-840a1521695db8fa5df0afbf24bcadb5e1dfc328f01456ecef065367bcacfd7c WatchSource:0}: Error finding container 840a1521695db8fa5df0afbf24bcadb5e1dfc328f01456ecef065367bcacfd7c: Status 404 returned error can't find the container with id 840a1521695db8fa5df0afbf24bcadb5e1dfc328f01456ecef065367bcacfd7c Oct 30 23:00:38.215888 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:38.215862 4335 generic.go:356] "Generic (PLEG): container finished" podID="c0e0cc0e-8e59-4023-a501-15caeeaeb538" containerID="831fe20dbab97f325988d7e757b7f7023cbb7be9093855e75aedd39dd8d35475" exitCode=0 Oct 30 23:00:38.216182 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:38.215903 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw" event={"ID":"c0e0cc0e-8e59-4023-a501-15caeeaeb538","Type":"ContainerDied","Data":"831fe20dbab97f325988d7e757b7f7023cbb7be9093855e75aedd39dd8d35475"} Oct 30 23:00:38.216182 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:38.215941 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw" event={"ID":"c0e0cc0e-8e59-4023-a501-15caeeaeb538","Type":"ContainerStarted","Data":"840a1521695db8fa5df0afbf24bcadb5e1dfc328f01456ecef065367bcacfd7c"} Oct 30 23:00:39.361779 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:39.361757 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw" Oct 30 23:00:39.396124 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:39.396100 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e0cc0e-8e59-4023-a501-15caeeaeb538-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0e0cc0e-8e59-4023-a501-15caeeaeb538-maas-api-openshift-ca\") pod \"c0e0cc0e-8e59-4023-a501-15caeeaeb538\" (UID: \"c0e0cc0e-8e59-4023-a501-15caeeaeb538\") " Oct 30 23:00:39.396248 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:39.396135 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0e0cc0e-8e59-4023-a501-15caeeaeb538-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0e0cc0e-8e59-4023-a501-15caeeaeb538-maas-api-xks-ca\") pod \"c0e0cc0e-8e59-4023-a501-15caeeaeb538\" (UID: \"c0e0cc0e-8e59-4023-a501-15caeeaeb538\") " Oct 30 23:00:39.396248 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:39.396237 4335 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e0cc0e-8e59-4023-a501-15caeeaeb538-kube-api-access-ltzrw\" (UniqueName: \"kubernetes.io/projected/c0e0cc0e-8e59-4023-a501-15caeeaeb538-kube-api-access-ltzrw\") pod \"c0e0cc0e-8e59-4023-a501-15caeeaeb538\" (UID: \"c0e0cc0e-8e59-4023-a501-15caeeaeb538\") " Oct 30 23:00:39.396493 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:39.396469 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e0cc0e-8e59-4023-a501-15caeeaeb538-maas-api-openshift-ca" pod "c0e0cc0e-8e59-4023-a501-15caeeaeb538" (UID: "c0e0cc0e-8e59-4023-a501-15caeeaeb538"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:00:39.396605 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:39.396591 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0e0cc0e-8e59-4023-a501-15caeeaeb538-maas-api-openshift-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:39.398287 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:39.398265 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0e0cc0e-8e59-4023-a501-15caeeaeb538-maas-api-xks-ca" pod "c0e0cc0e-8e59-4023-a501-15caeeaeb538" (UID: "c0e0cc0e-8e59-4023-a501-15caeeaeb538"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:00:39.398379 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:39.398357 4335 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e0cc0e-8e59-4023-a501-15caeeaeb538-kube-api-access-ltzrw" pod "c0e0cc0e-8e59-4023-a501-15caeeaeb538" (UID: "c0e0cc0e-8e59-4023-a501-15caeeaeb538"). InnerVolumeSpecName "kube-api-access-ltzrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:00:39.497980 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:39.497921 4335 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltzrw\" (UniqueName: \"kubernetes.io/projected/c0e0cc0e-8e59-4023-a501-15caeeaeb538-kube-api-access-ltzrw\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:39.497980 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:39.497943 4335 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0e0cc0e-8e59-4023-a501-15caeeaeb538-maas-api-xks-ca\") on node \"ip-10-0-131-193.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:00:40.229461 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:40.229431 4335 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw" Oct 30 23:00:40.229662 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:40.229472 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-btljw" event={"ID":"c0e0cc0e-8e59-4023-a501-15caeeaeb538","Type":"ContainerDied","Data":"840a1521695db8fa5df0afbf24bcadb5e1dfc328f01456ecef065367bcacfd7c"} Oct 30 23:00:40.229662 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:40.229523 4335 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="840a1521695db8fa5df0afbf24bcadb5e1dfc328f01456ecef065367bcacfd7c" Oct 30 23:00:49.364967 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:49.364932 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2gpxg"] Oct 30 23:00:53.047960 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:53.047926 4335 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg"] Oct 30 23:00:53.097085 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:53.097058 4335 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c666fa94-x76bg"] Oct 30 23:00:55.056259 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:55.056226 4335 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff60efe0-e305-4bd6-bce1-b6b7e5db6f00" path="/var/lib/kubelet/pods/ff60efe0-e305-4bd6-bce1-b6b7e5db6f00/volumes" Oct 30 23:00:55.319993 ip-10-0-131-193 kubenswrapper[4335]: E1030 23:00:55.319908 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 23:00:58.796951 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:58.796920 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-fc7c456c9-9z74t_e21a1a66-81c3-4e3f-a37b-5d2c6d799d2e/manager/0.log" Oct 30 23:00:59.150326 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:00:59.150297 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-k6246_e65cd6f7-2bfd-4b15-ba69-4175ab70a218/manager/0.log" Oct 30 23:01:00.329001 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:00.328972 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb56545f7-hrrcg_a8299996-476b-4d62-bd03-a25a4a46645f/manager/0.log" Oct 30 23:01:00.645452 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:00.645370 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xg5lh_2453d2d7-4fa2-4f97-91f8-ab58457daeb2/server/0.log" Oct 30 23:01:02.407857 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:02.407814 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-bkml9_ea1a5c9f-93c6-4407-8624-62627bc59533/revoke-keys/0.log" Oct 30 23:01:02.630123 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:02.630097 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-btljw_c0e0cc0e-8e59-4023-a501-15caeeaeb538/revoke-keys/0.log" Oct 30 23:01:02.853579 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:02.853553 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-8dxlp_abc66f6e-fa6c-41be-9221-4cbbcfa6db58/revoke-keys/0.log" Oct 30 23:01:05.047918 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:05.047892 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hwplz_8981dd9f-943f-4c22-88da-fb4395ff652b/manager/0.log" Oct 30 23:01:05.366218 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:05.366190 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-dlqz5_7cdc1aa2-31c1-4e01-9ad1-58baa27940fa/manager/0.log" Oct 30 23:01:05.842736 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:05.842710 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-dwsxg_8604f22a-5d96-4c12-b812-b9ac529c3dec/manager/0.log" Oct 30 23:01:06.219455 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:06.219396 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2gpxg_bc6e4759-93b2-49bd-ae8f-66a8ff627f02/limitador/0.log" Oct 30 23:01:06.445729 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:06.445705 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-shrdq_8d1c279d-f94f-4021-9bab-ed080adc78ca/manager/0.log" Oct 30 23:01:07.221938 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:07.221907 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654f2zkbf_80256cf8-6ab3-405b-9e40-afdc5de94b5b/istio-proxy/0.log" Oct 30 23:01:08.518592 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:08.518561 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-6bbbj_4e6addff-b99f-4e05-a17d-00bb0f88017c/istio-proxy/0.log" Oct 30 23:01:08.787116 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:08.787050 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-7nqtp_181f13d9-d1ab-43a9-b6a0-7f4a72f7632b/istio-proxy/0.log" Oct 30 23:01:09.825478 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:09.825449 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-jr7cw_bbf808e7-af04-425e-88db-2ebeb7e44240/istio-proxy/0.log" Oct 30 23:01:10.308631 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:10.308588 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-mjgkw_4ed2ae60-d2b4-4ce7-b48b-295a7eabe663/istio-proxy/0.log" Oct 30 23:01:10.743919 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:10.743847 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w_dec4cf36-264a-40d2-9a17-1d9e712d1238/istio-proxy/0.log" Oct 30 23:01:10.961190 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:10.961162 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qlsfm_bdd4f287-fcc7-4b03-b711-dcc0edab30e7/payload-pre-processing/0.log" Oct 30 23:01:11.184685 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:11.184647 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-dbz8f_21dcce2e-c36d-4de3-adb0-b514662b20e5/payload-processing/0.log" Oct 30 23:01:12.827270 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:12.827244 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-j7l8g_e472aa82-daee-470a-a544-bd40adc2073b/main/0.log" Oct 30 23:01:13.362419 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:13.362393 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm_2c054348-505a-4763-876b-5447d401e0ec/main/0.log" Oct 30 23:01:13.625311 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:13.625228 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-tpwc7_3e3c54c2-eb66-4b04-b6ef-55c1691873b9/main/0.log" Oct 30 23:01:14.101818 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:14.101789 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x_1023f325-f4c6-41ae-86f5-d0796101a380/main/0.log" Oct 30 23:01:24.035100 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:24.035066 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-fc7c456c9-9z74t_e21a1a66-81c3-4e3f-a37b-5d2c6d799d2e/manager/0.log" Oct 30 23:01:24.393761 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:24.393732 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-k6246_e65cd6f7-2bfd-4b15-ba69-4175ab70a218/manager/0.log" Oct 30 23:01:25.322546 ip-10-0-131-193 kubenswrapper[4335]: E1030 23:01:25.322510 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 23:01:25.327071 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:25.327051 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb56545f7-hrrcg_a8299996-476b-4d62-bd03-a25a4a46645f/manager/0.log" Oct 30 23:01:25.596277 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:25.596215 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xg5lh_2453d2d7-4fa2-4f97-91f8-ab58457daeb2/server/0.log" Oct 30 23:01:27.132306 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:27.132282 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-bkml9_ea1a5c9f-93c6-4407-8624-62627bc59533/revoke-keys/0.log" Oct 30 23:01:27.336001 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:27.335976 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-btljw_c0e0cc0e-8e59-4023-a501-15caeeaeb538/revoke-keys/0.log" Oct 30 23:01:27.550042 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:27.550013 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-8dxlp_abc66f6e-fa6c-41be-9221-4cbbcfa6db58/revoke-keys/0.log" Oct 30 23:01:29.400707 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:29.400672 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hwplz_8981dd9f-943f-4c22-88da-fb4395ff652b/manager/0.log" Oct 30 23:01:29.715598 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:29.715504 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-dlqz5_7cdc1aa2-31c1-4e01-9ad1-58baa27940fa/manager/0.log" Oct 30 23:01:30.170930 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:30.170902 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-dwsxg_8604f22a-5d96-4c12-b812-b9ac529c3dec/manager/0.log" Oct 30 23:01:30.537903 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:30.537833 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2gpxg_bc6e4759-93b2-49bd-ae8f-66a8ff627f02/limitador/0.log" Oct 30 23:01:30.765086 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:30.765055 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-shrdq_8d1c279d-f94f-4021-9bab-ed080adc78ca/manager/0.log" Oct 30 23:01:31.342796 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:31.342764 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654f2zkbf_80256cf8-6ab3-405b-9e40-afdc5de94b5b/istio-proxy/0.log" Oct 30 23:01:31.472776 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:31.472745 4335 scope.go:122] "RemoveContainer" containerID="f656ae526ea73cf628d38ed0583b86c205a634c08eb55a92e32d09a61389b438" Oct 30 23:01:31.483376 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:31.483354 4335 scope.go:122] "RemoveContainer" containerID="e2bab0a64c7e168aacdc0879583087e8286dcab8285edd51df3524dd9cf4f404" Oct 30 23:01:32.635058 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:32.635030 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-6bbbj_4e6addff-b99f-4e05-a17d-00bb0f88017c/istio-proxy/0.log" Oct 30 23:01:32.911637 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:32.911567 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-7nqtp_181f13d9-d1ab-43a9-b6a0-7f4a72f7632b/istio-proxy/0.log" Oct 30 23:01:33.961733 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:33.961700 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-jr7cw_bbf808e7-af04-425e-88db-2ebeb7e44240/istio-proxy/0.log" Oct 30 23:01:34.481932 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:34.481900 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-mjgkw_4ed2ae60-d2b4-4ce7-b48b-295a7eabe663/istio-proxy/0.log" Oct 30 23:01:34.963502 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:34.963401 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-95cb67c75-wbp5w_dec4cf36-264a-40d2-9a17-1d9e712d1238/istio-proxy/0.log" Oct 30 23:01:35.183044 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:35.183018 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qlsfm_bdd4f287-fcc7-4b03-b711-dcc0edab30e7/payload-pre-processing/0.log" Oct 30 23:01:35.392439 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:35.392414 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-dbz8f_21dcce2e-c36d-4de3-adb0-b514662b20e5/payload-processing/0.log" Oct 30 23:01:36.773106 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:36.773080 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-j7l8g_e472aa82-daee-470a-a544-bd40adc2073b/main/0.log" Oct 30 23:01:37.249016 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:37.248989 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kwclm_2c054348-505a-4763-876b-5447d401e0ec/main/0.log" Oct 30 23:01:37.521966 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:37.521893 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-tpwc7_3e3c54c2-eb66-4b04-b6ef-55c1691873b9/main/0.log" Oct 30 23:01:38.023949 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:01:38.023920 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-wrh4x_1023f325-f4c6-41ae-86f5-d0796101a380/main/0.log" Oct 30 23:01:55.320924 ip-10-0-131-193 kubenswrapper[4335]: E1030 23:01:55.320889 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 23:02:07.208368 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:07.208338 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb56545f7-hrrcg_a8299996-476b-4d62-bd03-a25a4a46645f/manager/0.log" Oct 30 23:02:07.769342 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:07.769312 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb56545f7-hrrcg_a8299996-476b-4d62-bd03-a25a4a46645f/manager/0.log" Oct 30 23:02:08.491477 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:08.491448 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-dbz8f_21dcce2e-c36d-4de3-adb0-b514662b20e5/payload-processing/0.log" Oct 30 23:02:08.714243 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:08.714214 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-dbz8f_21dcce2e-c36d-4de3-adb0-b514662b20e5/payload-processing/0.log" Oct 30 23:02:25.319944 ip-10-0-131-193 kubenswrapper[4335]: E1030 23:02:25.319907 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 23:02:31.513229 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:31.513192 4335 scope.go:122] "RemoveContainer" containerID="8b817e15118ef63470ddb76ecce2f7413e7a996f5fd1cea3a58dcd8c1ab1a24d" Oct 30 23:02:45.574022 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:45.573988 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb56545f7-hrrcg_a8299996-476b-4d62-bd03-a25a4a46645f/manager/0.log" Oct 30 23:02:46.133422 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:46.133395 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bb56545f7-hrrcg_a8299996-476b-4d62-bd03-a25a4a46645f/manager/0.log" Oct 30 23:02:46.883071 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:46.883043 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-dbz8f_21dcce2e-c36d-4de3-adb0-b514662b20e5/payload-processing/0.log" Oct 30 23:02:47.096582 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:47.096558 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-dbz8f_21dcce2e-c36d-4de3-adb0-b514662b20e5/payload-processing/0.log" Oct 30 23:02:47.471198 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:47.471172 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wz784/must-gather-zprg5"] Oct 30 23:02:47.471559 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:47.471547 4335 cpu_manager.go:414] "containerMap: removing container" podUID="c0e0cc0e-8e59-4023-a501-15caeeaeb538" containerName="revoke-keys" Oct 30 23:02:47.471606 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:47.471561 4335 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e0cc0e-8e59-4023-a501-15caeeaeb538" containerName="revoke-keys" Oct 30 23:02:47.471676 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:47.471658 4335 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e0cc0e-8e59-4023-a501-15caeeaeb538" containerName="revoke-keys" Oct 30 23:02:47.474909 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:47.474893 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wz784/must-gather-zprg5" Oct 30 23:02:47.527876 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:47.527854 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wz784\"/\"openshift-service-ca.crt\"" Oct 30 23:02:47.564176 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:47.564150 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/49c87a3d-0cc1-40ac-a98e-3f04d61cf2b9-must-gather-output\") pod \"must-gather-zprg5\" (UID: \"49c87a3d-0cc1-40ac-a98e-3f04d61cf2b9\") " pod="openshift-must-gather-wz784/must-gather-zprg5" Oct 30 23:02:47.564294 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:47.564258 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lsbf\" (UniqueName: \"kubernetes.io/projected/49c87a3d-0cc1-40ac-a98e-3f04d61cf2b9-kube-api-access-7lsbf\") pod \"must-gather-zprg5\" (UID: \"49c87a3d-0cc1-40ac-a98e-3f04d61cf2b9\") " pod="openshift-must-gather-wz784/must-gather-zprg5" Oct 30 23:02:47.577105 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:47.577087 4335 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wz784\"/\"kube-root-ca.crt\"" Oct 30 23:02:47.577420 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:47.577403 4335 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wz784\"/\"default-dockercfg-hz9zx\"" Oct 30 23:02:47.580348 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:47.580327 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wz784/must-gather-zprg5"] Oct 30 23:02:47.665182 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:47.665158 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lsbf\" (UniqueName: \"kubernetes.io/projected/49c87a3d-0cc1-40ac-a98e-3f04d61cf2b9-kube-api-access-7lsbf\") pod \"must-gather-zprg5\" (UID: \"49c87a3d-0cc1-40ac-a98e-3f04d61cf2b9\") " pod="openshift-must-gather-wz784/must-gather-zprg5" Oct 30 23:02:47.665346 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:47.665228 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/49c87a3d-0cc1-40ac-a98e-3f04d61cf2b9-must-gather-output\") pod \"must-gather-zprg5\" (UID: \"49c87a3d-0cc1-40ac-a98e-3f04d61cf2b9\") " pod="openshift-must-gather-wz784/must-gather-zprg5" Oct 30 23:02:47.665503 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:47.665487 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/49c87a3d-0cc1-40ac-a98e-3f04d61cf2b9-must-gather-output\") pod \"must-gather-zprg5\" (UID: \"49c87a3d-0cc1-40ac-a98e-3f04d61cf2b9\") " pod="openshift-must-gather-wz784/must-gather-zprg5" Oct 30 23:02:47.726742 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:47.726678 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lsbf\" (UniqueName: \"kubernetes.io/projected/49c87a3d-0cc1-40ac-a98e-3f04d61cf2b9-kube-api-access-7lsbf\") pod \"must-gather-zprg5\" (UID: \"49c87a3d-0cc1-40ac-a98e-3f04d61cf2b9\") " pod="openshift-must-gather-wz784/must-gather-zprg5" Oct 30 23:02:47.788754 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:47.788731 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wz784/must-gather-zprg5" Oct 30 23:02:48.162497 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:48.162464 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wz784/must-gather-zprg5"] Oct 30 23:02:48.164032 ip-10-0-131-193 kubenswrapper[4335]: W1030 23:02:48.164007 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49c87a3d_0cc1_40ac_a98e_3f04d61cf2b9.slice/crio-79c015f88a9e61c03751b3d11b2914843d739d6fa06347deaa9d7f7558c07fe4 WatchSource:0}: Error finding container 79c015f88a9e61c03751b3d11b2914843d739d6fa06347deaa9d7f7558c07fe4: Status 404 returned error can't find the container with id 79c015f88a9e61c03751b3d11b2914843d739d6fa06347deaa9d7f7558c07fe4 Oct 30 23:02:48.165883 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:48.165858 4335 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:02:49.058524 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:49.058491 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wz784/must-gather-zprg5" event={"ID":"49c87a3d-0cc1-40ac-a98e-3f04d61cf2b9","Type":"ContainerStarted","Data":"79c015f88a9e61c03751b3d11b2914843d739d6fa06347deaa9d7f7558c07fe4"} Oct 30 23:02:50.066100 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:50.066058 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wz784/must-gather-zprg5" event={"ID":"49c87a3d-0cc1-40ac-a98e-3f04d61cf2b9","Type":"ContainerStarted","Data":"4278f6bdf4a29cd8174a56a80a86b2f08e144d567ce0c091f3dec127ae3592b7"} Oct 30 23:02:50.066575 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:50.066109 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wz784/must-gather-zprg5" event={"ID":"49c87a3d-0cc1-40ac-a98e-3f04d61cf2b9","Type":"ContainerStarted","Data":"39047e0f26593a84ed471b51ec1316285489d31963b0e6faadd4d05074d425b8"} Oct 30 23:02:50.183289 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:50.183239 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wz784/must-gather-zprg5" podStartSLOduration=1.94215369 podStartE2EDuration="3.183223027s" totalImagesPullingTime="1.241069337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:50.181217132 +0000 UTC m=+4703.281648865" watchObservedRunningTime="2026-10-30 23:02:50.183223027 +0000 UTC m=+4703.283654760" Oct 30 23:02:55.323894 ip-10-0-131-193 kubenswrapper[4335]: E1030 23:02:55.323855 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 23:02:57.250435 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:02:57.248568 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8mr69_f73329a3-7290-44a1-a72e-1fe72c0d0da5/global-pull-secret-syncer/0.log" Oct 30 23:03:00.328049 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:00.328013 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lpx2r_08df8d3b-f392-438a-81d5-3ee04ee2cd8b/konnectivity-agent/0.log" Oct 30 23:03:01.563772 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:01.563740 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-193.us-west-1.compute.internal_8b76742bc66466f5dd9bc77bf5370e40/haproxy/0.log" Oct 30 23:03:06.112670 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.112631 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hwplz_8981dd9f-943f-4c22-88da-fb4395ff652b/manager/0.log" Oct 30 23:03:06.541045 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.540977 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-dlqz5_7cdc1aa2-31c1-4e01-9ad1-58baa27940fa/manager/0.log" Oct 30 23:03:06.655937 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.655904 4335 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd"] Oct 30 23:03:06.664406 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.664384 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:06.716636 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.716125 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd"] Oct 30 23:03:06.757321 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.757287 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eadce641-7415-4a4a-b48c-ba2177314e96-lib-modules\") pod \"perf-node-gather-daemonset-r7hqd\" (UID: \"eadce641-7415-4a4a-b48c-ba2177314e96\") " pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:06.757510 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.757382 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k45ft\" (UniqueName: \"kubernetes.io/projected/eadce641-7415-4a4a-b48c-ba2177314e96-kube-api-access-k45ft\") pod \"perf-node-gather-daemonset-r7hqd\" (UID: \"eadce641-7415-4a4a-b48c-ba2177314e96\") " pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:06.757510 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.757410 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eadce641-7415-4a4a-b48c-ba2177314e96-sys\") pod \"perf-node-gather-daemonset-r7hqd\" (UID: \"eadce641-7415-4a4a-b48c-ba2177314e96\") " pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:06.757628 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.757596 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eadce641-7415-4a4a-b48c-ba2177314e96-podres\") pod \"perf-node-gather-daemonset-r7hqd\" (UID: \"eadce641-7415-4a4a-b48c-ba2177314e96\") " pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:06.757695 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.757661 4335 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eadce641-7415-4a4a-b48c-ba2177314e96-proc\") pod \"perf-node-gather-daemonset-r7hqd\" (UID: \"eadce641-7415-4a4a-b48c-ba2177314e96\") " pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:06.858937 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.858903 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k45ft\" (UniqueName: \"kubernetes.io/projected/eadce641-7415-4a4a-b48c-ba2177314e96-kube-api-access-k45ft\") pod \"perf-node-gather-daemonset-r7hqd\" (UID: \"eadce641-7415-4a4a-b48c-ba2177314e96\") " pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:06.859141 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.858945 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eadce641-7415-4a4a-b48c-ba2177314e96-sys\") pod \"perf-node-gather-daemonset-r7hqd\" (UID: \"eadce641-7415-4a4a-b48c-ba2177314e96\") " pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:06.859141 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.859051 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eadce641-7415-4a4a-b48c-ba2177314e96-podres\") pod \"perf-node-gather-daemonset-r7hqd\" (UID: \"eadce641-7415-4a4a-b48c-ba2177314e96\") " pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:06.859141 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.859080 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eadce641-7415-4a4a-b48c-ba2177314e96-proc\") pod \"perf-node-gather-daemonset-r7hqd\" (UID: \"eadce641-7415-4a4a-b48c-ba2177314e96\") " pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:06.859141 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.859098 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eadce641-7415-4a4a-b48c-ba2177314e96-sys\") pod \"perf-node-gather-daemonset-r7hqd\" (UID: \"eadce641-7415-4a4a-b48c-ba2177314e96\") " pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:06.859141 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.859117 4335 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eadce641-7415-4a4a-b48c-ba2177314e96-lib-modules\") pod \"perf-node-gather-daemonset-r7hqd\" (UID: \"eadce641-7415-4a4a-b48c-ba2177314e96\") " pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:06.859400 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.859167 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eadce641-7415-4a4a-b48c-ba2177314e96-proc\") pod \"perf-node-gather-daemonset-r7hqd\" (UID: \"eadce641-7415-4a4a-b48c-ba2177314e96\") " pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:06.859400 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.859261 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eadce641-7415-4a4a-b48c-ba2177314e96-podres\") pod \"perf-node-gather-daemonset-r7hqd\" (UID: \"eadce641-7415-4a4a-b48c-ba2177314e96\") " pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:06.859400 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.859277 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eadce641-7415-4a4a-b48c-ba2177314e96-lib-modules\") pod \"perf-node-gather-daemonset-r7hqd\" (UID: \"eadce641-7415-4a4a-b48c-ba2177314e96\") " pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:06.919035 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.919007 4335 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k45ft\" (UniqueName: \"kubernetes.io/projected/eadce641-7415-4a4a-b48c-ba2177314e96-kube-api-access-k45ft\") pod \"perf-node-gather-daemonset-r7hqd\" (UID: \"eadce641-7415-4a4a-b48c-ba2177314e96\") " pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:06.977658 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:06.977607 4335 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:07.353066 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:07.353039 4335 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd"] Oct 30 23:03:07.355747 ip-10-0-131-193 kubenswrapper[4335]: W1030 23:03:07.355716 4335 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeadce641_7415_4a4a_b48c_ba2177314e96.slice/crio-dd90354c01b0ec09910d2366df6d7603333d6df73ee71cdc64b71f0e1379fcd4 WatchSource:0}: Error finding container dd90354c01b0ec09910d2366df6d7603333d6df73ee71cdc64b71f0e1379fcd4: Status 404 returned error can't find the container with id dd90354c01b0ec09910d2366df6d7603333d6df73ee71cdc64b71f0e1379fcd4 Oct 30 23:03:07.410099 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:07.410067 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-dwsxg_8604f22a-5d96-4c12-b812-b9ac529c3dec/manager/0.log" Oct 30 23:03:07.860061 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:07.860035 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2gpxg_bc6e4759-93b2-49bd-ae8f-66a8ff627f02/limitador/0.log" Oct 30 23:03:08.229923 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:08.229814 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" event={"ID":"eadce641-7415-4a4a-b48c-ba2177314e96","Type":"ContainerStarted","Data":"ea81c4d1b9479659f8e9c52bd7f471599c29d3b54a48fb644e3039d6eec86e6f"} Oct 30 23:03:08.229923 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:08.229868 4335 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" event={"ID":"eadce641-7415-4a4a-b48c-ba2177314e96","Type":"ContainerStarted","Data":"dd90354c01b0ec09910d2366df6d7603333d6df73ee71cdc64b71f0e1379fcd4"} Oct 30 23:03:08.230149 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:08.229966 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:08.327340 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:08.327269 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-shrdq_8d1c279d-f94f-4021-9bab-ed080adc78ca/manager/0.log" Oct 30 23:03:08.346559 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:08.346514 4335 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" podStartSLOduration=2.346500129 podStartE2EDuration="2.346500129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:03:08.344456574 +0000 UTC m=+4721.444888330" watchObservedRunningTime="2026-10-30 23:03:08.346500129 +0000 UTC m=+4721.446931861" Oct 30 23:03:14.250336 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:14.250307 4335 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wz784/perf-node-gather-daemonset-r7hqd" Oct 30 23:03:19.071988 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:19.071921 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z7ndn_e295034c-effb-4a8d-ad28-d2068c6c1616/node-exporter/0.log" Oct 30 23:03:19.491292 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:19.491213 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z7ndn_e295034c-effb-4a8d-ad28-d2068c6c1616/kube-rbac-proxy/0.log" Oct 30 23:03:19.908183 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:19.908154 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z7ndn_e295034c-effb-4a8d-ad28-d2068c6c1616/init-textfile/0.log" Oct 30 23:03:25.323997 ip-10-0-131-193 kubenswrapper[4335]: E1030 23:03:25.323953 4335 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f8pkp" containerName="extractor" Oct 30 23:03:25.688308 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:25.688210 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-t268b_80badb63-a1f9-437a-aeb0-d0ca556b99e0/prometheus-operator-admission-webhook/0.log" Oct 30 23:03:29.493571 ip-10-0-131-193 kubenswrapper[4335]: I1030 23:03:29.493534 4335 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654f2zkbf_80256cf8-6ab3-405b-9e40-afdc5de94b5b/istio-proxy/0.log"